Monitoring Guide

Monitoring Guide

This guide covers monitoring Bifrost using Prometheus metrics, Grafana dashboards, and health checks.

Prometheus Metrics

Bifrost exposes metrics in Prometheus format at the configured metrics endpoint.

Configuration

metrics:
  enabled: true
  listen: ":7090"
  path: "/metrics"

Scrape Configuration

Add to your prometheus.yml:

scrape_configs:
  - job_name: 'bifrost-server'
    static_configs:
      - targets: ['bifrost-server:7090']
    scrape_interval: 15s

  - job_name: 'bifrost-client'
    static_configs:
      - targets: ['bifrost-client:7090']
    scrape_interval: 15s

Available Metrics

These are the exact series exported by the server (see internal/metrics/prometheus.go). In addition, the standard go_* and process_* collectors are registered automatically.

Connection Metrics

Metric	Type	Labels	Description
bifrost_connections_total	Counter	protocol, backend	Total connections handled
bifrost_connections_active	Gauge	protocol, backend	Current active connections
bifrost_connection_duration_seconds	Histogram	protocol, backend	Connection duration

backend on the active gauge is always empty. A connection is accepted before routing picks a backend, so bifrost_connections_active is only ever tracked at protocol scope and its backend label is the empty string. The resolved backend name is recorded on bifrost_connections_total and bifrost_connection_duration_seconds once the connection finishes. For a live per-backend breakdown use bifrost_backend_connections instead.

Request Metrics

Metric	Type	Labels	Description
bifrost_requests_total	Counter	protocol, method, status	Total requests
bifrost_request_duration_seconds	Histogram	protocol, method	Request duration
bifrost_request_size_bytes	Histogram	protocol	Request body size
bifrost_response_size_bytes	Histogram	protocol	Response body size

Backend Metrics

Metric	Type	Labels	Description
bifrost_backend_health	Gauge	backend, type	Backend health (1=healthy, 0=unhealthy)
bifrost_backend_connections	Gauge	backend	Active connections per backend
bifrost_backend_latency_seconds	Histogram	backend	Backend health-check latency
bifrost_backend_errors_total	Counter	backend, error_type	Total backend errors

Traffic Metrics

Metric	Type	Labels	Description
bifrost_bytes_sent_total	Counter	backend	Total bytes sent
bifrost_bytes_received_total	Counter	backend	Total bytes received

Rate Limiting Metrics

Metric	Type	Labels	Description
bifrost_rate_limit_hits_total	Counter	type	Total rate limit hits

Auth Metrics

Metric	Type	Labels	Description
bifrost_auth_attempts_total	Counter	method	Total authentication attempts
bifrost_auth_failures_total	Counter	method, reason	Total authentication failures

Cache Metrics

Exported only when cache.enabled: true — the cache collectors are registered together with the cache manager at startup, so on a server with caching off the bifrost_cache_* series are absent rather than zero.

Metric	Type	Labels	Description
bifrost_cache_hits_total	Counter	domain	Cache hits, by request host
bifrost_cache_misses_total	Counter	domain, reason	Cache misses; reason is no_rule, not_found or expired
bifrost_cache_bytes_served_total	Counter	source	Bytes served, source="cache" or source="origin"
bifrost_cache_bytes_cached_total	Counter	—	Bytes written into the cache
bifrost_cache_storage_size_bytes	Gauge	tier	Current storage size per tier
bifrost_cache_storage_entries	Gauge	tier	Current entry count per tier
bifrost_cache_storage_usage_percent	Gauge	tier	Storage usage as a percentage of max_size
bifrost_cache_evictions_total	Counter	tier, reason	Evictions; reason is size or ttl
bifrost_cache_operation_duration_seconds	Histogram	operation	Latency of get and put operations
bifrost_cache_active_rules	Gauge	—	Number of active cache rules
bifrost_cache_active_presets	Gauge	—	Number of enabled cache presets

The tier label is memory or disk for those storage types; a tiered cache reports both. bifrost_cache_storage_usage_percent is only set when the tier has a non-zero max_size.

System Metrics

Metric	Type	Description
bifrost_uptime_seconds	Gauge	Server uptime in seconds
bifrost_goroutines	Gauge	Number of goroutines

Note: There is no bifrost_memory_bytes gauge — use the standard process_resident_memory_bytes (from the process collector) for memory usage.

Example Queries

# Request rate per second
rate(bifrost_requests_total[5m])

# Average request duration
rate(bifrost_request_duration_seconds_sum[5m]) / rate(bifrost_request_duration_seconds_count[5m])

# Error rate (share of requests with a 5xx status)
sum(rate(bifrost_requests_total{status=~"5.."}[5m])) / sum(rate(bifrost_requests_total[5m]))

# Active connections by backend
bifrost_backend_connections

# Throughput (MB/s) sent to backends
rate(bifrost_bytes_sent_total[5m]) / 1024 / 1024

# Unhealthy backends
bifrost_backend_health == 0

# Backend error rate
sum(rate(bifrost_backend_errors_total[5m])) by (backend)

# Cache hit ratio (requires cache.enabled: true)
sum(rate(bifrost_cache_hits_total[5m]))
  / (sum(rate(bifrost_cache_hits_total[5m])) + sum(rate(bifrost_cache_misses_total[5m])))

# Share of bytes served from cache rather than origin
sum(rate(bifrost_cache_bytes_served_total{source="cache"}[5m]))
  / sum(rate(bifrost_cache_bytes_served_total[5m]))

# Disk cache fill level
bifrost_cache_storage_usage_percent{tier="disk"}

# Resident memory (from the process collector)
process_resident_memory_bytes

---

Grafana Dashboards

Docker Compose Setup

The included Docker Compose file starts Grafana with Prometheus:

cd docker
docker-compose up -d grafana prometheus

Access Grafana at http://localhost:3000 (default: admin/admin).

Add Prometheus Data Source

1. Go to Configuration → Data Sources
2. Add data source → Prometheus
3. URL: http://prometheus:7090
4. Save & Test

Dashboard Panels

Overview Panel

{
  "title": "Active Connections",
  "type": "stat",
  "targets": [{
    "expr": "bifrost_connections_active",
    "legendFormat": "Connections"
  }]
}

Request Rate Panel

{
  "title": "Request Rate",
  "type": "graph",
  "targets": [{
    "expr": "rate(bifrost_requests_total[5m])",
    "legendFormat": "{{protocol}} {{method}} - {{status}}"
  }]
}

Backend Health Panel

{
  "title": "Backend Health",
  "type": "table",
  "targets": [{
    "expr": "bifrost_backend_health",
    "format": "table",
    "instant": true
  }]
}

Latency Heatmap

{
  "title": "Request Latency",
  "type": "heatmap",
  "targets": [{
    "expr": "rate(bifrost_request_duration_seconds_bucket[5m])",
    "format": "heatmap"
  }]
}

---

Health Checks

HTTP Health Endpoint

curl http://localhost:7082/api/v1/health

Response:

{
  "status": "healthy",
  "time": "2024-01-15T10:30:00Z"
}

Status values:

• healthy - All backends healthy
• degraded - Some backends unhealthy

Backend Health

curl http://localhost:7082/api/v1/backends

Response:

[
  {
    "name": "direct",
    "type": "direct",
    "healthy": true,
    "stats": {
      "total_connections": 1234,
      "active_connections": 5,
      "bytes_sent": 1048576,
      "bytes_received": 2097152
    }
  }
]

Docker Health Check

healthcheck:
  test: ["CMD", "wget", "-q", "--spider", "http://localhost:7090/metrics"]
  interval: 30s
  timeout: 5s
  retries: 3
  start_period: 5s

Kubernetes Probes

livenessProbe:
  httpGet:
    path: /api/v1/health
    port: 8082
  initialDelaySeconds: 5
  periodSeconds: 10

readinessProbe:
  httpGet:
    path: /api/v1/health
    port: 8082
  initialDelaySeconds: 5
  periodSeconds: 5

---

Alerting

Prometheus Alerting Rules

Create alerts.yml:

groups:
  - name: bifrost
    rules:
      # High error rate (share of requests returning a 5xx status)
      - alert: BifrostHighErrorRate
        expr: sum(rate(bifrost_requests_total{status=~"5.."}[5m])) / sum(rate(bifrost_requests_total[5m])) > 0.05
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High error rate on Bifrost"
          description: "Error rate is {{ $value | humanizePercentage }}"

      # Backend down
      - alert: BifrostBackendDown
        expr: bifrost_backend_health == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Bifrost backend {{ $labels.backend }} is down"

      # High latency
      - alert: BifrostHighLatency
        expr: histogram_quantile(0.95, rate(bifrost_request_duration_seconds_bucket[5m])) > 5
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High latency on Bifrost"
          description: "P95 latency is {{ $value | humanizeDuration }}"

      # No connections
      - alert: BifrostNoConnections
        expr: bifrost_connections_active == 0
        for: 10m
        labels:
          severity: info
        annotations:
          summary: "No active connections on Bifrost"

      # High connection count
      - alert: BifrostHighConnections
        expr: bifrost_connections_active > 10000
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High connection count"
          description: "{{ $value }} active connections"

Grafana Alerts

1. Edit a panel
2. Go to Alert tab
3. Create alert rule
4. Set conditions and notifications

---

Logging

Structured Logging

logging:
  level: info
  format: json
  output: stdout      # stdout, stderr, or a file path
  time_format: "2006-01-02T15:04:05.000Z07:00"
  max_size_mb: 100    # File outputs only: rotate above this size (<= 0 disables rotation)
  max_backups: 14     # Rotated files to retain (<= 0 keeps all)

max_size_mb and max_backups drive Bifrost’s built-in size-based rotation and apply only when output is a file path — see Log Rotation in the deployment guide.

Log Levels

Level	Description
debug	Verbose debugging information
info	Normal operational messages
warn	Warning conditions
error	Error conditions

Log Aggregation

Loki (with Docker)

# docker-compose.yml
services:
  loki:
    image: grafana/loki:2.9.0
    ports:
      - "3100:3100"
    volumes:
      - loki-data:/loki

  bifrost-server:
    logging:
      driver: loki
      options:
        loki-url: "http://localhost:3100/loki/api/v1/push"
        labels: "app=bifrost,service=server"

Elasticsearch

logging:
  format: json
  output: stdout

Use Filebeat to ship logs to Elasticsearch:

# filebeat.yml
filebeat.inputs:
  - type: container
    paths:
      - '/var/lib/docker/containers/*/*.log'

output.elasticsearch:
  hosts: ["elasticsearch:9200"]

---

Best Practices

Metrics

1. Scrape interval: 15-30 seconds for most use cases
2. Retention: Keep at least 15 days of metrics
3. Labels: Avoid high-cardinality labels (e.g., user IDs)

Alerting

1. Start simple: Begin with basic alerts, add more as needed
2. Avoid alert fatigue: Only alert on actionable issues
3. Document runbooks: Link alerts to troubleshooting guides

Dashboards

1. Overview first: Start with high-level health metrics
2. Drill-down: Allow navigation to detailed views
3. Time ranges: Support common ranges (1h, 6h, 24h, 7d)

Logging

1. Structured logs: Use JSON format for parsing
2. Correlation IDs: Include request IDs for tracing
3. Log rotation: Prevent disk space issues