LLM gateway
One endpoint for every model. We handle the routing.
Point your apps at the UnitVane AI gateway instead of each provider. Every request is routed to the right deployment, falls back when a provider throttles, stays within its team's budget, and arrives in your tokenomics with its exact cost.
Every team calls models its own way
Keys everywhere:
Provider keys live in a dozen repos, with no budget and no owner attached to them.Throttling breaks features:
When a deployment returns 429, the feature fails instead of moving to capacity that is free.Reserved capacity sits idle:
PTUs are paid for every hour, while some apps still send traffic to pay-as-you-go.Cost arrives a month late:
The bill shows a total per deployment, not which app, team or prompt spent it.
Drop-in endpoint
Change the base URL, keep your code
The gateway speaks the OpenAI API for chat, responses and embeddings, and the Anthropic Messages API. Apps send a route name such as chat-default instead of a vendor model, so you can change models and providers without a deploy.
- OpenAI and Anthropic SDKs work unchanged
- Streaming, tools and JSON mode passed through
- Route names decouple apps from vendors
import osfrom openai import OpenAIclient = OpenAI( base_url="https://gateway.unitvane.com/v1", api_key=os.environ["KUBEON_AI_KEY"], # a team key, not a provider key)resp = client.chat.completions.create( model="chat-default", # a route name, not a vendor model messages=[{"role": "user", "content": "Summarize this ticket in two lines."}],)print(resp.choices[0].message.content)Routing and fallback
PTU first, pay-as-you-go next, another provider if both are busy
Each route lists its targets in order. Requests use reserved capacity until it reaches the utilization you set, spill over to pay-as-you-go, and fall back to another provider on 429s, 5xx errors or timeouts. Other strategies pick the cheapest or fastest target, or split traffic by weight.
- Priority, lowest cost, lowest latency or weighted
- Retries with backoff, per-target timeouts
- Region pinning for data residency
apiVersion: ai.unitvane.com/v1kind: Routemetadata: name: chat-defaultspec: strategy: priority # priority | lowest-cost | lowest-latency | weighted targets: - provider: azure-openai deployment: chat-prod # gpt-4.1, Data Zone, 100 PTU until: utilization: 90% # PTU first, then spill over - provider: azure-openai deployment: chat-paygo # gpt-4.1, Global pay-as-you-go - provider: bedrock model: anthropic.claude-sonnet region: us-east-1 on: [429, 5xx, timeout] # fallback only retries: { attempts: 2, backoff: exponential } timeoutSeconds: 20 cache: { mode: exact, ttl: 1h } redact: [email, phone, credit_card] residency: us maxOutputTokens: 4096Team keys
A key per team, with its own models, budget and rate limit
Issue a gateway key for each team or app. Each key carries the routes it may call, a monthly budget that warns or stops at 100%, and limits in tokens and requests per minute. Provider credentials stay in the gateway, never in application code.
- Soft warnings or hard stops at budget
- Tokens-per-minute and requests-per-minute limits
- Keys mapped to product, team and tenant
curl https://acme.kubeon.io/api/ai/keys \ -H "Authorization: Bearer $KUBEON_ADMIN_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "name": "support-bot", "team": "Support assistant", "product": "Customer support", "routes": ["chat-default", "embed"], "budget": { "monthlyUsd": 50000, "action": "hard-stop" }, "limits": { "tokensPerMinute": 2000000, "requestsPerMinute": 3000 } }'Per-request metering
Every request with its tokens, route and cost
The gateway records input, output and cached tokens, the target that served the request, latency, fallbacks and the cost at your contracted price. Bodies are not logged unless you turn logging on, cached responses expire with the route's TTL, and email addresses, phone numbers and card numbers are masked before a request leaves the gateway.
- Cost per request, key, route and model
- Exact-match cache with cached tokens counted
- Export to OpenTelemetry
curl https://gateway.unitvane.com/v1/chat/completions \ -H "Authorization: Bearer $KUBEON_AI_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "chat-default", "messages": [{"role": "user", "content": "Summarize this ticket in two lines."}] }'# Response headers# x-kubeon-route: chat-default/1 (azure-openai chat-prod, ptu)# x-kubeon-tokens: prompt=1840 completion=96 cached=1024# x-kubeon-cost-usd: 0.0000 (covered by reserved PTU)Why AI spend needs an owner
Where it runs, and what it keeps
Managed by us, in the region you choose
UnitVane runs the gateway in the region you choose, so you have nothing to operate. Every request takes the same path:
| Step | What happens | What is kept |
|---|---|---|
| App to gateway | Your app sends the request with its team key over HTTPS, to the gateway in the region you choose. | Nothing |
| Routing | The gateway reads the request in memory and picks a target: the PTU deployment, then pay-as-you-go, then a fallback, all within the route's residency setting. | Nothing |
| Masking | On routes with masking on, email addresses, phone numbers and card numbers are replaced before the request leaves the gateway. Names and free text are not detected. | Nothing |
| Provider | The request goes to your own provider account with your credentials, stored encrypted. The provider bills you directly. | Under your provider's terms |
| Response | The response returns to your app. On routes with the cache on, it is kept for identical requests. | Cached response, until the route's TTL ends |
| Metering | Tokens, route, target, latency, status and cost are recorded for every request in your workspace. | Request metadata, 90 days by default |
| Logging | Prompt and response bodies are written to the log only on routes where you turn logging on. | Bodies, only with logging on |
Provider credentials are used only to call the provider you configured for each route.
A gateway you do not have to build
Routing strategies
Priority, lowest cost, lowest latency and weighted splits.
Fallback and retries
Move to the next target on 429, 5xx or timeout.
PTU-first spillover
Use reserved capacity before paying per token.
Team keys
Routes, budgets and limits per team or app.
Response cache
Exact-match caching with a TTL per route.
Email, phone and card masking
Emails, phone numbers and card numbers masked before sending.
Data residency
Pin a route to the regions your data must stay in.
Observability
Per-request cost, latency and route, exported to OpenTelemetry.
LLM gateway: your questions answered
UnitVane AI
Keep exploring
Evaluate LLM gateway with your team
Bring your platform and FinOps priorities. We will walk through the data, controls and evidence for your evaluation.