Rolling out fail-closed mode safely
Moving from fail-open to fail-closed is worth doing carefully: the whole point is to block on failure, so a bad threshold can block real traffic. Here's a low-risk path.
1. Start in monitor mode
Run enforcement in observe-only mode first — decisions are recorded but nothing is blocked:
ENFORCEMENT_MODE=monitor
THREAT_AUTO_BLOCK=falseLet it run against production traffic long enough to see a representative sample.
2. Validate on staging
Replay traffic against staging with the load harnesses that ship with the platform, and watch your block-vs-allow rates on the Prometheus metrics and the dashboard threats view. Confirm the allow rate for known-good traffic stays where you expect.
3. Tighten path-by-path
Enforcement is configurable per path, so enable fail-closed on the highest-risk paths first — the AI Gateway and MCP routes — then move outward once each one looks clean. Flip THREAT_AUTO_BLOCK=true and ENFORCEMENT_MODE=block only for paths you've validated.
4. Keep rollback one step away
- ▸Any single path can be reverted without a global change.
- ▸Blue-green deployment slots let you roll back with zero downtime.
- ▸Caching is a performance layer, not a correctness one — a cache miss always falls back safely.
Sustained high-throughput latency is not independently load-proven. Before you fail-close a hot path, validate timeout headroom under realistic multi-worker load so a slow check doesn't start denying good traffic.
The semantic tier's false-positive rate rises on out-of-distribution benign traffic. If your traffic is unusual, watch the false-block rate closely while you tighten.