Monitoring Guide

Monitoring Guide

This guide covers monitoring Bifrost using Prometheus metrics, Grafana dashboards, and health checks.

Prometheus Metrics

Bifrost exposes metrics in Prometheus format at the configured metrics endpoint.

Configuration

metrics:
  enabled: true
  listen: ":7090"
  path: "/metrics"

Scrape Configuration

Add to your prometheus.yml:

scrape_configs:
  - job_name: 'bifrost-server'
    static_configs:
      - targets: ['bifrost-server:7090']
    scrape_interval: 15s

  - job_name: 'bifrost-client'
    static_configs:
      - targets: ['bifrost-client:7090']
    scrape_interval: 15s

Available Metrics

These are the exact series exported by the server (see internal/metrics/prometheus.go). In addition, the standard go_* and process_* collectors are registered automatically.

Connection Metrics

Metric	Type	Labels	Description
bifrost_connections_total	Counter	protocol, backend	Total connections handled
bifrost_connections_active	Gauge	protocol, backend	Current active connections
bifrost_connection_duration_seconds	Histogram	protocol, backend	Connection duration

Request Metrics

Metric	Type	Labels	Description
bifrost_requests_total	Counter	protocol, method, status	Total requests
bifrost_request_duration_seconds	Histogram	protocol, method	Request duration
bifrost_request_size_bytes	Histogram	protocol	Request body size
bifrost_response_size_bytes	Histogram	protocol	Response body size

Backend Metrics

Metric	Type	Labels	Description
bifrost_backend_health	Gauge	backend, type	Backend health (1=healthy, 0=unhealthy)
bifrost_backend_connections	Gauge	backend	Active connections per backend
bifrost_backend_latency_seconds	Histogram	backend	Backend health-check latency
bifrost_backend_errors_total	Counter	backend, error_type	Total backend errors

Traffic Metrics

Metric	Type	Labels	Description
bifrost_bytes_sent_total	Counter	backend	Total bytes sent
bifrost_bytes_received_total	Counter	backend	Total bytes received

Rate Limiting Metrics

Metric	Type	Labels	Description
bifrost_rate_limit_hits_total	Counter	type	Total rate limit hits

Auth Metrics

Metric	Type	Labels	Description
bifrost_auth_attempts_total	Counter	method	Total authentication attempts
bifrost_auth_failures_total	Counter	method, reason	Total authentication failures

System Metrics

Metric	Type	Description
bifrost_uptime_seconds	Gauge	Server uptime in seconds
bifrost_goroutines	Gauge	Number of goroutines

Note: There is no bifrost_memory_bytes gauge — use the standard process_resident_memory_bytes (from the process collector) for memory usage. Prometheus-format cache metrics (bifrost_cache_*) are not exported; cache statistics are available through the REST API (/api/v1/cache/stats) and the Web UI instead.

Example Queries

# Request rate per second
rate(bifrost_requests_total[5m])

# Average request duration
rate(bifrost_request_duration_seconds_sum[5m]) / rate(bifrost_request_duration_seconds_count[5m])

# Error rate (share of requests with a 5xx status)
sum(rate(bifrost_requests_total{status=~"5.."}[5m])) / sum(rate(bifrost_requests_total[5m]))

# Active connections by backend
bifrost_backend_connections

# Throughput (MB/s) sent to backends
rate(bifrost_bytes_sent_total[5m]) / 1024 / 1024

# Unhealthy backends
bifrost_backend_health == 0

# Backend error rate
sum(rate(bifrost_backend_errors_total[5m])) by (backend)

# Resident memory (from the process collector)
process_resident_memory_bytes

---

Grafana Dashboards

Docker Compose Setup

The included Docker Compose file starts Grafana with Prometheus:

cd docker
docker-compose up -d grafana prometheus

Access Grafana at http://localhost:3000 (default: admin/admin).

Add Prometheus Data Source

1. Go to Configuration → Data Sources
2. Add data source → Prometheus
3. URL: http://prometheus:7090
4. Save & Test

Dashboard Panels

Overview Panel

{
  "title": "Active Connections",
  "type": "stat",
  "targets": [{
    "expr": "bifrost_connections_active",
    "legendFormat": "Connections"
  }]
}

Request Rate Panel

{
  "title": "Request Rate",
  "type": "graph",
  "targets": [{
    "expr": "rate(bifrost_requests_total[5m])",
    "legendFormat": "{{protocol}} {{method}} - {{status}}"
  }]
}

Backend Health Panel

{
  "title": "Backend Health",
  "type": "table",
  "targets": [{
    "expr": "bifrost_backend_health",
    "format": "table",
    "instant": true
  }]
}

Latency Heatmap

{
  "title": "Request Latency",
  "type": "heatmap",
  "targets": [{
    "expr": "rate(bifrost_request_duration_seconds_bucket[5m])",
    "format": "heatmap"
  }]
}

---

Health Checks

HTTP Health Endpoint

curl http://localhost:7082/api/v1/health

Response:

{
  "status": "healthy",
  "time": "2024-01-15T10:30:00Z"
}

Status values:

• healthy - All backends healthy
• degraded - Some backends unhealthy

Backend Health

curl http://localhost:7082/api/v1/backends

Response:

[
  {
    "name": "direct",
    "type": "direct",
    "healthy": true,
    "stats": {
      "total_connections": 1234,
      "active_connections": 5,
      "bytes_sent": 1048576,
      "bytes_received": 2097152
    }
  }
]

Docker Health Check

healthcheck:
  test: ["CMD", "wget", "-q", "--spider", "http://localhost:7090/metrics"]
  interval: 30s
  timeout: 5s
  retries: 3
  start_period: 5s

Kubernetes Probes

livenessProbe:
  httpGet:
    path: /api/v1/health
    port: 8082
  initialDelaySeconds: 5
  periodSeconds: 10

readinessProbe:
  httpGet:
    path: /api/v1/health
    port: 8082
  initialDelaySeconds: 5
  periodSeconds: 5

---

Alerting

Prometheus Alerting Rules

Create alerts.yml:

groups:
  - name: bifrost
    rules:
      # High error rate (share of requests returning a 5xx status)
      - alert: BifrostHighErrorRate
        expr: sum(rate(bifrost_requests_total{status=~"5.."}[5m])) / sum(rate(bifrost_requests_total[5m])) > 0.05
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High error rate on Bifrost"
          description: "Error rate is {{ $value | humanizePercentage }}"

      # Backend down
      - alert: BifrostBackendDown
        expr: bifrost_backend_health == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Bifrost backend {{ $labels.backend }} is down"

      # High latency
      - alert: BifrostHighLatency
        expr: histogram_quantile(0.95, rate(bifrost_request_duration_seconds_bucket[5m])) > 5
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High latency on Bifrost"
          description: "P95 latency is {{ $value | humanizeDuration }}"

      # No connections
      - alert: BifrostNoConnections
        expr: bifrost_connections_active == 0
        for: 10m
        labels:
          severity: info
        annotations:
          summary: "No active connections on Bifrost"

      # High connection count
      - alert: BifrostHighConnections
        expr: bifrost_connections_active > 10000
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High connection count"
          description: "{{ $value }} active connections"

Grafana Alerts

1. Edit a panel
2. Go to Alert tab
3. Create alert rule
4. Set conditions and notifications

---

Logging

Structured Logging

logging:
  level: info
  format: json
  output: stdout  # or file path

Log Levels

Level	Description
debug	Verbose debugging information
info	Normal operational messages
warn	Warning conditions
error	Error conditions

Log Aggregation

Loki (with Docker)

# docker-compose.yml
services:
  loki:
    image: grafana/loki:2.9.0
    ports:
      - "3100:3100"
    volumes:
      - loki-data:/loki

  bifrost-server:
    logging:
      driver: loki
      options:
        loki-url: "http://localhost:3100/loki/api/v1/push"
        labels: "app=bifrost,service=server"

Elasticsearch

logging:
  format: json
  output: stdout

Use Filebeat to ship logs to Elasticsearch:

# filebeat.yml
filebeat.inputs:
  - type: container
    paths:
      - '/var/lib/docker/containers/*/*.log'

output.elasticsearch:
  hosts: ["elasticsearch:9200"]

---

Best Practices

Metrics

1. Scrape interval: 15-30 seconds for most use cases
2. Retention: Keep at least 15 days of metrics
3. Labels: Avoid high-cardinality labels (e.g., user IDs)

Alerting

1. Start simple: Begin with basic alerts, add more as needed
2. Avoid alert fatigue: Only alert on actionable issues
3. Document runbooks: Link alerts to troubleshooting guides

Dashboards

1. Overview first: Start with high-level health metrics
2. Drill-down: Allow navigation to detailed views
3. Time ranges: Support common ranges (1h, 6h, 24h, 7d)

Logging

1. Structured logs: Use JSON format for parsing
2. Correlation IDs: Include request IDs for tracing
3. Log rotation: Prevent disk space issues