We publish what is measured and what is not. Measured: single-user latency on the production node, and the cost of the full security chain on the completion route on one worker of a development workstation. No production-node load figure exists yet. Here is what G8KEPR adds to a request, and the scripts to measure it on your own hardware.
Context matters. The only under-load figures on this page are the completion-route run below: one worker on a shared development workstation, against an instant stub model. They are not production capacity. The older committed load runs recorded mostly rate-limit rejections, so their requests-per-second counts rejections, not capacity (loadtests/RUNBOOK_throughput.md). The k6 and Locust scripts are in the repo (tests/load, loadtests) — run them on your own hardware.
Measured single-user latency, the held-out detection test, and one-worker throughput of the completion route on a development workstation (details below).
These thresholds are defined in tests/load/k6-ci.js. No CI job runs them today (the load-test workflows are disabled), and the last committed run failed all 14 of its thresholds. They are the bar a run has to clear, not a result.
Sanity check — single-user baseline, tightest thresholds
Sustained-load threshold
Burst threshold
Published targets from tests/load/benchmark_results.json
GET /healthReadiness probe
POST /api/keys/validateAPI key authentication
GET /api/gateway/proxyGateway passthrough
GET /api/compliance/frameworksCompliance data fetch
GET /api/audit-logsAudit log retrieval
POST /api/threat-intelligence/analyzeFull AI threat scan
These are targets, not measurements. On the completion route detection runs in line, before the reply is returned; its measured cost is in the security-tax section below.
The detection engine is a four-tier cascade. Cheap tiers run first; expensive tiers escalate only when confidence is low. The fast path stays under 25ms on every request.
~340 pre-compiled regex rules evaluated on every request: prompt injection, jailbreak, role hijacking, PII extraction and context manipulation. High-confidence (>0.9) hits block immediately.
SentenceTransformer cosine similarity vs. known injection vectors. Catches semantic equivalents that bypass literal regex. Lazy-loaded model.
Intent classification gate. Plus base64 / URL / hex decode-then-rescan, NFKD normalization, character-split detection, Shannon entropy.
High-uncertainty inputs only. Routed via the AI Gateway to an LLM provider. Async — fail-open on timeout to avoid blocking response.
< 25ms regex-only · < 40ms with ML scoring
Scored on a frozen 2,580-sample held-out set it was never trained on: 0.788 recall, 0.857 precision and a 3.12% false-positive rate. The test fails the build on any regression. On a separate set of 50 adversarial-evasion + 50 benign prompts, 45 of 50 evasions (90%) are caught.
Every request goes through only the steps it needs. On the completion route the detectors, guardrails and response checks run in line, before the reply is returned; their measured total is in the security-tax section below.
In-memory JWT signature check + Redis GET for key scope
Redis INCR with sliding window — single round-trip
In-memory state check — no I/O
Redis semantic cache hit — no model call needed
Pattern matching + optional ML scoring. Runs in line on the completion route: a refused prompt never reaches the model.
Async PostgreSQL insert — does not add to request latency
Pass-through total: ~15–30ms added overhead for JWT validation + rate limit + circuit breaker. No more.
Threat analysis: runs in line on the completion route, before the reply is returned, so a refused prompt never reaches the model. The time it costs is measured in the security-tax section below.
Performance is a first-class design constraint, not an afterthought. These are the specific choices that give G8KEPR its overhead profile.
All endpoints are async/await — no thread blocking on I/O. Single worker can handle hundreds of concurrent connections without spawning threads.
Identical concurrent requests to Redis are coalesced — only one cache miss fires even under burst. Semantic cache prevents re-running ML models for repeated prompt patterns.
PostgreSQL connections are pooled via asyncpg — no per-request connect overhead. Pool sizing is tuned per deployment based on DB capacity.
API responses are Brotli-compressed at the edge, reducing wire bytes by ~30% vs gzip. HTTP/3 eliminates head-of-line blocking for clients that support it.
These aren't tunables that need configuring after deployment — they're structural decisions that keep latency flat as load scales. None of them are present in a generic API gateway plus third-party AI guardrail stack.
Cheap tiers run first. 99% of traffic terminates at Tier 1 regex (<1ms). Only ambiguous inputs escalate to embeddings (5-15ms), NLI (<15ms), or LLM (rare). p99 stays <25ms regex-only, <40ms with ML.
modules/threat_detection/SHA-256 fingerprint per system prompt. After 2 identical prompts, auto-injects cache_control:ephemeral. Break-even at 2 calls (38% off). 88% off at 10 calls. Zero SDK changes.
gateway/cache_optimizer.pyStatistical baseline per provider per hour-of-day, not static thresholds. 4 rolling windows (1m / 5m / 15m / 1hr). Progressive recovery 10/25/50/100%. Off by default; enable with ADAPTIVE_CB_ENABLED.
gateway/router.pyIdentical concurrent requests are coalesced — only one cache miss fires under burst. Semantic cache prevents re-running ML for repeated patterns. Cache miss is always safe; every cache has a documented fallback path.
11 Redis caches · TTL fail-safePer-request overhead measured on the production node: one client sending 150 sequential requests to 127.0.0.1 through the full middleware stack and authentication (an unrouted path, so no handler runs), after 15 warm-up requests. No CI job re-runs it.
This is the HTTP path up to authentication, not detection time. Reproduce on the host with python3 scripts/perf/measure_request_latency.py -n 150 --warmup 15.
Overhead is the latency of a completion through G8KEPR minus the same request sent straight to the same stub model: 360 requests each at 2 per second, every security control at its shipped default, no rate limit raised. It covers everything the gateway adds (authentication, rate limits, the request-side detectors and guardrails, routing, the verification engine and the response filter), and the stub answers in about a millisecond, so a real model's own latency comes on top. Throughput is 8 clients sending back to back for 120 seconds. Measured on one uvicorn worker of a shared development workstation (AMD Ryzen 5 3600XT), not production hardware; a repeat run gave 189.36 / 283.98 ms and 4.825 completions/s. Artifact and method: docs/benchmarks/gateway-security-tax/.
Production-node load: not measured. The committed 50-user Locust run recorded 5,233 failures in 6,147 requests, almost all of them rate-limit rejections, so its requests-per-second counts rejections rather than capacity. The method is in loadtests/RUNBOOK_throughput.md.
Self-hosting removes the network hop to a remote gateway. What is left is processing, and that is measured above.
Eleven distinct Redis caches with explicit TTLs and explicit fallbacks. Cache miss is always safe — Redis provides performance, not correctness. Every cache has a documented degraded path.
300sIn-memory LRU300sIn-memory dict300sDB query30sIn-memory LRU1 hourLLM API call5 minRouting algorithmUntil expiryFail-closed (reject)15 minFlow failsPer windowDegraded (warn)1 hourDB query60sDB queryPublic paths: max-age=60, stale-while-revalidate=300 · Private paths: max-age=30, must-revalidate · 304 Not Modified on hit
Singleton with max 50 connections · 5s connect timeout · TLS for remote hosts · background ping every 30s · Redis ops < 1ms typical
All test scripts are in the repository. No black-box benchmarks — every number on this page is reproducible. Run against the demo API or your own self-hosted deployment.
tests/load/k6-ci.jsCI gate — smoke + 50 VU load + 200 VU stress
tests/load/k6-baseline.jsAuthoritative baseline — 5 key endpoints at 50 VU
tests/load/k6-full.jsFull stress test — 0→500 VU ramp (staging only)
backend/tests/performance/benchmark_guard_performance.pySemanticGuard cache performance (pytest)
Production note: Self-hosted on production hardware (4+ vCPU, 8GB RAM) will significantly outperform these demo numbers. Contact us for production benchmark guidance.
All k6 scripts are in the repo. Run them against the demo API or your own deployment. If you want a production sizing conversation, we are happy to help.