Route, cache, guardrail, and audit every LLM call — across 14 providers
Most gateways route. G8KEPR also auto-injects Anthropic prompt caching (38% cost cut at 2 calls, 88% at 10), can adapt circuit breakers with Z-score baselines instead of static thresholds (opt-in — off by default, ADAPTIVE_CB_ENABLED=true to turn it on), and stamps an EU AI Act risk class on every completion. BYOK with AES-256-GCM — your keys, your provider rates, your audit trail.
14 providers, one API. Add a custom endpoint and we route through that too:
The missing infrastructure layer for production LLM applications
Production AI apps need multiple LLM providers for reliability, cost optimization, and feature coverage. But managing them means juggling separate keys, billing accounts, rate limits, and error handling scattered across every service that calls an LLM.
Most teams hard-code provider-specific logic across services. Every switch is a code change, a redeploy, and a test pass. Cost tracking is manual. When a provider degrades, you find out from your users — not your monitoring.
One unified API that intelligently routes to the best LLM provider
client.chat.completions.create(model="auto")Single API call with model="auto" - no provider-specific code
✓ Routed to Claude • Cost-optimized • 145ms response • Failover to GPT-4 if unavailable
Ordered provider list — first healthy provider wins. Predictable routing with deterministic fallback order.
✓ Claude → GPT-4 → GeminiCheapest provider that meets the requested model capability. Gemini Flash ($0.50/M) for simple tasks, Claude ($3/M) for reasoning.
✓ Save up to 92%Lowest p95 latency from historical tracking. Real-time metrics, not stale benchmarks. Critical for user-facing chat.
✓ Sub-200ms p95Distribute load evenly across healthy providers. Sidesteps single-provider rate-limit ceilings during traffic bursts.
✓ No 429s under burstClassify the prompt (CODING, CREATIVE, ANALYSIS, GENERAL) and route to the model that excels at that intent.
✓ Best-fit model per callFive capabilities that exist in the platform — not in OpenAI's SDK, not in LangChain, not in a one-file proxy you wrote on a Friday afternoon.
Tracks system-prompt hashes per org. After 2 identical, auto-injects Anthropic cache_control:ephemeral. Zero user code changes.
gateway/cache_optimizer.pyOpt-in (off by default; ADAPTIVE_CB_ENABLED=true to turn it on). When on: statistical baselines per provider per hour-of-day. Trips when failure rate > mean + 3σ. Progressive recovery 10/25/50/100%.
gateway/router.pyRule-based toxicity, bias, topic-block, PII, prompt-injection, regex, content-length and rate-limit policy types. PII, prompt-injection, toxicity, bias and topic-block are seeded as advisory (log-only) defaults; regex, content-length and rate-limit run once an admin creates them. Block / redact / warn / log per policy. Every violation logged.
ai_guardrail_policiesEvery response carries an EU AI Act risk-class header, set once per deployment (default limited_risk). Per-model classification is not yet available.
X-AI-Risk-ClassKeys encrypted at rest, decrypted only into process memory. Never written to Redis or logs. Spend capped by the organization budget.
EncryptionServiceSystem prompts are usually 80%+ of the token cost on every call — and they're identical every time. G8KEPR fingerprints them with SHA-256, and after observing two identical prompts, automatically injects Anthropic's cache_control: ephemeral directive. No SDK changes, no flag flipping — it just happens.
Static thresholds break when traffic patterns shift. G8KEPR's breaker learns a per-endpoint baseline with time-of-day awareness — it knows that a 2% failure rate is normal at 3 a.m. and abnormal at 3 p.m. Opt-in: enable it with ADAPTIVE_CB_ENABLED.
Trip at 50% failure rate. Doesn't know that this provider runs hot at 3 a.m. anyway. False trips during normal off-peak. Misses gradual degradation that stays under the threshold.
Hour-specific baseline. 2% failures at 3 a.m. is normal — won't trip. 2% failures at peak is a spike — trips immediately. Catches what threshold breakers miss.
How G8KEPR customers save 60-90% on LLM costs with intelligent routing
See exactly what you're spending across all LLM providers in one dashboard
| Provider | Model | Requests | Tokens | Rate | Cost |
|---|---|---|---|---|---|
C Claude | 3.5 Sonnet | 12,456 | 2.3M | $3/M | $6.90 |
G OpenAI | GPT-4 Turbo | 1,234 | 0.8M | $30/M | $24.00 |
G Google | Gemini Flash | 45,123 | 8.2M | $0.50/M | $4.10 |
| Total | 58,813 | 11.3M | Avg $3.10/M | $35.00 | |
Everything you need to manage multi-LLM applications in production
Bring your own keys for the built-in providers. Encrypted at rest, decrypted only into process memory — never written to Redis or logs. Used only for requests from your organization, within its gateway budget.
✓ POST /api/gateway/keys · llm_api_keysConfigure ordered fallback chains: Claude → GPT-4 → Gemini. Automatic re-route on 5xx, 429, latency-SLO miss, or health-check failure. Health tracked per provider per hour.
✓ Per-provider exponential backoffTag requests with user_id, team_id, project_id. Per-org / per-user / per-provider / per-model rollup. Z-score anomaly detection fires Slack and email when spend spikes.
✓ gateway_usage_logs · cost_budgetsRule-based toxicity, bias, topic-block, PII, prompt-injection, regex, content-length and rate-limit policy types — evaluated on the prompt before it leaves the gateway once a policy of that type exists. PII, prompt-injection, toxicity, bias and topic-block are seeded as advisory defaults; the other three are created by an admin. Block / redact / warn / log per policy.
✓ ai_guardrail_policies · ai_guardrail_violationsA PII guardrail policy scans each prompt before it leaves the gateway for SSNs, card numbers, phone numbers, emails, IP addresses, dates of birth, API keys, JWTs, passwords, titled names, ICD-10 codes, ZIP codes and crypto wallets — then blocks, redacts with [TYPE] placeholders, or records, per policy.
✓ ai_guardrail_policies · guardrail type piiLLM_HARD_MAX_TOKENS=16,384 enforced on every provider, regardless of user input. A caller can't pass max_tokens=999999 to drain the budget in one shot.
✓ Cost-amplification attack defenseEvery response carries X-AI-Risk-Class, set once per deployment through AI_RISK_CLASS_DEFAULT (default limited_risk). Per-model classification is not yet available.
✓ X-AI-Risk-Class · AI_RISK_CLASS_DEFAULTBYOI custom endpoints and the 13 first-party providers that call over HTTP build their client through one factory that installs SSRFProtectedTransport: each request is refused if its hostname resolves to a loopback, RFC1918, link-local or cloud-metadata (169.254.169.254) address, with IPv4-mapped IPv6 normalized first. AWS Bedrock signs with SigV4 through boto3 to AWS endpoints and does not use this transport. Ollama and BYOI therefore work only against a publicly routable server, not localhost or a private network.
✓ gateway/providers/_ssrf_client.py · gateway/http_client.pyDrop-in replacement — change base URL, route to any of 14 providers. No code changes to application logic. Same chat-completions schema, same streaming, same function-call format.
✓ One-line integrationEverything you need to know about multi-LLM routing
Need help setting up multi-LLM routing?
Talk to our AI Gateway experts →Every completion carries the EU AI Act risk class and a full audit-log entry. Mappings are pre-built — auditors get exports, not spreadsheets.
Subject to independent audit and attestation. G8KEPR provides the technical controls and evidence — your auditor issues the certification.
Auto prompt caching, opt-in adaptive circuit breakers (off by default), eight guardrail policy types, EU AI Act headers, and BYOK with AES-256-GCM — across 14 providers, with no per-token markup.
No credit card required • BYOK - no markup • Cancel anytime