Prometheus Metrics: Production-Grade Observability

Native Prometheus metrics are coming to G8KEPR in Q1 2025, providing production-grade observability with pre-built Grafana dashboards and OpenTelemetry support for distributed tracing.

Why Prometheus Metrics Matter

You can't secure what you can't measure. Prometheus metrics give you real-time visibility into:

Available Metrics

G8KEPR exposes 40+ Prometheus metrics at /metrics:

HTTP Request Metrics

# HELP http_requests_total Total HTTP requests processed
# TYPE http_requests_total counter
http_requests_total{method="GET",endpoint="/api/users",status="200"} 45231
http_requests_total{method="POST",endpoint="/api/auth/login",status="401"} 127

# HELP http_request_duration_seconds HTTP request latency
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{endpoint="/api/users",le="0.005"} 40000
http_request_duration_seconds_bucket{endpoint="/api/users",le="0.01"} 44500
http_request_duration_seconds_bucket{endpoint="/api/users",le="0.025"} 45000
http_request_duration_seconds_bucket{endpoint="/api/users",le="+Inf"} 45231
http_request_duration_seconds_sum{endpoint="/api/users"} 234.5
http_request_duration_seconds_count{endpoint="/api/users"} 45231

Security Metrics

# HELP malicious_requests_blocked_total Malicious requests blocked
# TYPE malicious_requests_blocked_total counter
malicious_requests_blocked_total{threat_type="sql_injection"} 234
malicious_requests_blocked_total{threat_type="xss"} 45
malicious_requests_blocked_total{threat_type="path_traversal"} 12

# HELP rate_limit_exceeded_total Rate limit violations
# TYPE rate_limit_exceeded_total counter
rate_limit_exceeded_total{endpoint="/api/search",user="user_123"} 5
rate_limit_exceeded_total{endpoint="/api/auth/login",ip="1.2.3.4"} 15

Upstream Service Metrics

# HELP upstream_requests_total Requests to upstream services
# TYPE upstream_requests_total counter
upstream_requests_total{service="user-api",status="200"} 12000
upstream_requests_total{service="user-api",status="500"} 5

# HELP upstream_duration_seconds Upstream service latency
# TYPE upstream_duration_seconds histogram
upstream_duration_seconds_sum{service="user-api"} 450.2
upstream_duration_seconds_count{service="user-api"} 12005

Grafana Dashboards

We provide pre-built Grafana dashboards you can import instantly:

Dashboard ID: 17823 (grafana.com)

Dashboard ID: 17824 (grafana.com)

Dashboard ID: 17825 (grafana.com)

Prometheus Configuration

Add G8KEPR as a scrape target in your Prometheus config:

# prometheus.yml
scrape_configs:
  - job_name: 'g8kepr'
    scrape_interval: 15s
    static_configs:
      - targets: ['g8kepr:8000']
    metrics_path: /metrics

    # If G8KEPR is behind auth:
    basic_auth:
      username: prometheus
      password: ${METRICS_PASSWORD}

  # For Kubernetes:
  - job_name: 'g8kepr-k8s'
    kubernetes_sd_configs:
      - role: pod
        namespaces:
          names: [default]
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        action: keep
        regex: g8kepr

Alerting Rules

Example Prometheus alerts for G8KEPR:

# alerts.yml
groups:
  - name: g8kepr_security
    interval: 30s
    rules:
      - alert: HighMaliciousTraffic
        expr: rate(malicious_requests_blocked_total[5m]) > 100
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High rate of malicious requests detected"
          description: "{{ $value }} malicious requests/sec in last 5min"

      - alert: BruteForceAttack
        expr: rate(rate_limit_exceeded_total{endpoint="/api/auth/login"}[1m]) > 10
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Potential brute force attack on login endpoint"

  - name: g8kepr_performance
    interval: 30s
    rules:
      - alert: HighErrorRate
        expr: rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m]) > 0.05
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Error rate above 5%"

      - alert: HighLatency
        expr: histogram_quantile(0.99, http_request_duration_seconds_bucket) > 1.0
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "P99 latency above 1 second"

OpenTelemetry Integration

For distributed tracing across microservices:

# config.yaml
observability:
  tracing:
    enabled: true
    exporter: otlp
    endpoint: http://jaeger:4318

    # Sample rate (0.0 to 1.0)
    sample_rate: 0.1  # Trace 10% of requests

    # Always trace these endpoints:
    always_sample:
      - /api/auth/*
      - /api/admin/*

  # Trace propagation headers
  propagate_headers:
    - traceparent  # W3C Trace Context
    - tracestate
    - x-request-id  # Custom correlation ID

Metrics Cardinality Control

Prevent metrics explosion from high-cardinality labels:

# config.yaml
metrics:
  # Limit unique values per label
  cardinality_limits:
    endpoint: 1000  # Max 1000 unique endpoints
    user_id: 10000  # Max 10000 unique users
    ip: 100000  # Max 100k unique IPs

  # Drop metrics for low-traffic endpoints
  endpoint_threshold: 10  # Must have 10+ req/min

  # Aggregate dynamic paths:
  path_normalization:
    enabled: true
    # /api/users/123 → /api/users/:id
    # /api/posts/456 → /api/posts/:id

Export Metrics to Other Systems

# Use Datadog agent with
# openmetrics check:
instances:
  - prometheus_url: http://g8kepr:8000/metrics
    namespace: g8kepr
    metrics: ["*"]

# Use Prometheus remote write:
remote_write:
  - url: https://metric-api.newrelic.com/prometheus/v1/write?prometheus_server=g8kepr
    bearer_token: YOUR_LICENSE_KEY

# Use CloudWatch agent:
  [[inputs.prometheus]]
    urls = ["http://g8kepr:8000/metrics"]
  [[outputs.cloudwatch]]
    namespace = "G8KEPR"

# Use Splunk OpenTelemetry Collector:
receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: g8kepr
          static_configs:
            - targets: [g8kepr:8000]

Real-Time Metrics in Dashboard

If you use the G8KEPR cloud dashboard, metrics are automatically synced every 60 seconds without any Prometheus setup: