Authentication Troubleshooting

Authentication Troubleshooting

This guide covers authentication-related issues and their solutions across all supported authentication modes.

Quick Diagnostics

# Test authentication with credentials
curl -x http://user:password@localhost:7080 https://httpbin.org/ip -v

# Check for auth-related errors in logs
journalctl -u bifrost-server | grep -i "auth\|unauthorized\|forbidden"

# Verify auth configuration
bifrost-server validate -c config.yaml

---

Proxy Authentication Required (407)

Symptoms: Clients receive HTTP 407 response.

Cause 1: Missing Credentials

Credentials are not being sent with the request.

Diagnosis:

# Test without credentials (should fail)
curl -x http://localhost:7080 https://example.com -v 2>&1 | grep "407"

# Test with credentials
curl -x http://user:password@localhost:7080 https://example.com -v

Solution:

Configure your client to send credentials:

# curl
curl -x http://user:password@localhost:7080 https://example.com

# wget
wget -e use_proxy=yes -e http_proxy=http://user:password@localhost:7080 https://example.com

# Environment variable
export http_proxy="http://user:password@localhost:7080"
export https_proxy="http://user:password@localhost:7080"

Cause 2: Incorrect Password Hash

The stored password hash doesn’t match the provided password.

Diagnosis:

# Verify hash format (should be bcrypt)
grep password_hash config.yaml

Solution:

Generate a correct bcrypt hash:

# Using htpasswd
htpasswd -nbBC 12 "" "mypassword" | cut -d: -f2

# Using Python
python3 -c "import bcrypt; print(bcrypt.hashpw(b'mypassword', bcrypt.gensalt(12)).decode())"

# Using Go
go run -mod=mod github.com/rennerdo30/bifrost-proxy/tools/hashpw mypassword

Update configuration:

auth:
  providers:
    - name: native
      type: native
      enabled: true
      priority: 1
      config:
        users:
          - username: myuser
            password_hash: "$2a$12$correcthashhere..."

Cause 3: Special Characters in Password

Passwords with special characters need URL encoding.

Solution:

URL encode special characters:

Character	Encoded
@	%40
:	%3A
!	%21
#	%23
$	%24
&	%26
+	%2B
/	%2F
?	%3F

# Password "p@ss:word!" becomes:
curl -x http://user:p%40ss%3Aword%21@localhost:7080 https://example.com

---

Native Authentication Issues

User Not Found

Symptoms: “user not found” error in logs.

Solution:

Verify user exists in configuration:

auth:
  providers:
    - name: native
      type: native
      enabled: true
      priority: 1
      config:
        users:
          - username: myuser  # Case-sensitive!
            password_hash: "$2a$12$..."

Account Disabled

Symptoms: “account disabled” error.

Solution:

Enable the user account:

auth:
  providers:
    - name: native
      type: native
      enabled: true
      priority: 1
      config:
        users:
          - username: myuser
            password_hash: "$2a$12$..."
            disabled: false  # Must be false or omitted

---

LDAP Authentication Issues

Connection Failed

Symptoms: “connection refused” or “timeout” errors for LDAP.

Diagnosis:

# Test LDAP connectivity
ldapsearch -x -H ldap://ldap.example.com:389 -D "cn=admin,dc=example,dc=com" -W -b "dc=example,dc=com"

# Test network connectivity
nc -zv ldap.example.com 389
telnet ldap.example.com 389

Solution:

1. Verify LDAP server is reachable
2. Check firewall rules
3. Verify URL format:

Standard LDAP:

auth:
  providers:
    - name: ldap
      type: ldap
      enabled: true
      priority: 1
      config:
        url: "ldap://ldap.example.com:389"
        base_dn: "dc=example,dc=com"

LDAP over TLS:

auth:
  providers:
    - name: ldap
      type: ldap
      enabled: true
      priority: 1
      config:
        url: "ldaps://ldap.example.com:636"
        base_dn: "dc=example,dc=com"

Both url and base_dn are required — the provider is rejected at startup if either is missing.

Bind Failed

Symptoms: “invalid credentials” error during LDAP bind.

Diagnosis:

# Test bind credentials
ldapsearch -x -H ldap://ldap.example.com -D "cn=service,dc=example,dc=com" -w 'password' -b "dc=example,dc=com"

Solution:

Verify bind DN and password:

auth:
  providers:
    - name: ldap
      type: ldap
      enabled: true
      priority: 1
      config:
        url: "ldap://ldap.example.com:389"
        base_dn: "dc=example,dc=com"
        bind_dn: "cn=service,dc=example,dc=com"
        bind_password: "${LDAP_BIND_PASSWORD}"  # Use environment variable

User Not Found in LDAP

Symptoms: “user not found” even though user exists in LDAP.

Diagnosis:

# Search for user manually
ldapsearch -x -H ldap://ldap.example.com -D "cn=admin,dc=example,dc=com" -W -b "dc=example,dc=com" "(uid=myuser)"

Solution:

Adjust user filter to match your LDAP schema:

auth:
  providers:
    - name: ldap
      type: ldap
      enabled: true
      priority: 1
      config:
        url: "ldap://ldap.example.com:389"
        base_dn: "dc=example,dc=com"
        # Unix-style LDAP (posixAccount)
        user_filter: "(uid=%s)"
        # For Active Directory:      user_filter: "(sAMAccountName=%s)"
        # For email-based lookup:    user_filter: "(mail=%s)"

Pick exactly one user_filter. A YAML mapping cannot define the same key twice — listing several variants in one block makes the configuration file unparseable and the server refuses to start.

Group Membership Check Failing

Symptoms: User authenticates but is denied due to group requirements.

Diagnosis:

# Check user's group membership
ldapsearch -x -H ldap://ldap.example.com -D "cn=admin,dc=example,dc=com" -W -b "dc=example,dc=com" "(memberUid=myuser)"

Solution:

Adjust group filter or remove group requirement:

auth:
  providers:
    - name: ldap
      type: ldap
      enabled: true
      priority: 1
      config:
        url: "ldap://ldap.example.com:389"
        base_dn: "dc=example,dc=com"
        # For groupOfNames (member DN)
        group_filter: "(member=uid=%s,ou=users,dc=example,dc=com)"
        # For posixGroup (member UID): group_filter: "(memberUid=%s)"

        # Require specific group (omit or leave empty to drop the requirement)
        require_group: "cn=proxy-users,ou=groups,dc=example,dc=com"

As with user_filter, only one group_filter may be present per provider.

LDAP TLS Issues

Symptoms: TLS handshake errors.

Diagnosis:

# Test TLS connection
openssl s_client -connect ldap.example.com:636

# Check certificate
openssl s_client -connect ldap.example.com:636 -showcerts

Solution:

auth:
  providers:
    - name: ldap
      type: ldap
      enabled: true
      priority: 1
      config:
        url: "ldaps://ldap.example.com:636"
        base_dn: "dc=example,dc=com"
        tls: true
        # For testing only (trusts any server certificate):
        insecure_skip_verify: true  # Never use in production!

---

System Authentication Issues

Platform Not Supported

Symptoms: “system authentication is not supported on Windows” error.

Solution:

System authentication only works on Linux and macOS. Use an alternative:

On Windows, use native authentication instead:

auth:
  providers:
    - name: native
      type: native
      enabled: true
      priority: 1
      config:
        users:
          - username: admin
            password_hash: "$2a$12$..."

Or LDAP against a domain controller for Active Directory:

auth:
  providers:
    - name: ldap
      type: ldap
      enabled: true
      priority: 1
      config:
        url: "ldap://your-dc.domain.com:389"
        base_dn: "dc=domain,dc=com"
        user_filter: "(sAMAccountName=%s)"

Each option is a complete configuration file on its own — auth: may appear only once. To run both providers, list them side by side under the same providers: array and order them with priority.

Every login rejected / “PAM password validation is not compiled in” (Linux)

Symptoms: All system-auth logins fail and the log shows system auth: PAM password validation is not compiled in on Linux; failing closed.

Cause: The default (and Docker) binaries are built without the pam build tag and with cgo disabled, so Linux system auth fails closed.

Solution: Rebuild with the PAM backend enabled:

# Requires libpam headers (libpam0g-dev / pam-devel)
CGO_ENABLED=1 go build -tags pam -o bifrost-server ./cmd/server

Permission Denied (Linux)

Symptoms: A PAM-enabled build fails with permission errors.

Solution:

Bifrost needs appropriate permissions to use PAM:

# Run as root (not recommended)
sudo bifrost-server -c config.yaml

# Or configure PAM permissions
# Check /etc/pam.d/login or create /etc/pam.d/bifrost

User/Group Restrictions

Symptoms: User authenticates but is denied.

Solution:

Check allowed users/groups configuration:

auth:
  providers:
    - name: system
      type: system
      enabled: true
      priority: 1
      config:
        allowed_users:
          - alice
          - bob
        allowed_groups:
          - admin
          - proxy-users

---

OAuth/OIDC Issues

Invalid Client ID/Secret

Symptoms: “invalid_client” error from OAuth provider.

Diagnosis:

# Test OAuth token endpoint manually
curl -X POST https://auth.example.com/oauth/token \
  -d "client_id=YOUR_CLIENT_ID" \
  -d "client_secret=YOUR_CLIENT_SECRET" \
  -d "grant_type=client_credentials"

Solution:

Verify credentials with your OAuth provider:

auth:
  providers:
    - name: oauth
      type: oauth
      enabled: true
      priority: 1
      config:
        client_id: "${OAUTH_CLIENT_ID}"
        client_secret: "${OAUTH_CLIENT_SECRET}"
        issuer_url: "https://auth.example.com"

client_id is required. So is at least one endpoint — introspect_url, userinfo_url, or an issuer_url from which Bifrost discovers them.

Redirect URI Mismatch

Symptoms: “redirect_uri_mismatch” error while obtaining a token.

Cause: The Bifrost OAuth provider validates bearer tokens by introspection or via the userinfo endpoint — it never runs an authorization-code flow, so it has no redirect URI and accepts no redirect_url setting. A redirect_uri_mismatch therefore comes from the application that obtained the token, not from Bifrost.

Solution:

Fix the redirect URI in the client application and in its registration with the identity provider. On the Bifrost side, only the introspection endpoint and credentials matter:

auth:
  providers:
    - name: oauth
      type: oauth
      enabled: true
      priority: 1
      config:
        client_id: "${OAUTH_CLIENT_ID}"
        client_secret: "${OAUTH_CLIENT_SECRET}"
        introspect_url: "https://auth.example.com/oauth/introspect"

Token Validation Failed

Symptoms: Token is accepted by provider but rejected by Bifrost.

Diagnosis:

# Decode JWT token (if applicable)
echo "your.jwt.token" | cut -d. -f2 | base64 -d | jq

Solution:

Verify issuer and audience settings:

auth:
  providers:
    - name: oauth
      type: oauth
      enabled: true
      priority: 1
      config:
        client_id: "${OAUTH_CLIENT_ID}"
        client_secret: "${OAUTH_CLIENT_SECRET}"
        issuer_url: "https://auth.example.com"  # Must match token's 'iss' claim
        required_claims:
          aud: "bifrost-proxy"

---

JWT Authentication Issues

Invalid Signature

Symptoms: “signature verification failed” error.

Diagnosis:

# Check token signature at jwt.io
# Or decode locally
echo "your.jwt.token" | cut -d. -f2 | base64 -d | jq

Solution:

1. Using JWKS (recommended):

   auth:
     providers:
       - name: jwt
         type: jwt
         enabled: true
         priority: 1
         config:
           jwks_url: "https://auth.example.com/.well-known/jwks.json"

2. Using a static public key (RSA/EC tokens, RS*/ES* algorithms):

   auth:
     providers:
       - name: jwt
         type: jwt
         enabled: true
         priority: 1
         config:
           public_key_pem: |
             -----BEGIN PUBLIC KEY-----
             MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8A...
             -----END PUBLIC KEY-----

3. Using a shared secret (HMAC tokens, HS256/HS384/HS512):

   auth:
     providers:
       - name: jwt
         type: jwt
         enabled: true
         priority: 1
         config:
           hmac_secret: "${JWT_HMAC_SECRET}"
           algorithms:
             - HS256

Exactly one of jwks_url, public_key_pem, or hmac_secret must be present — without one the provider is rejected at startup. HMAC and asymmetric key material are never interchangeable: listing an HS* algorithm without hmac_secret is rejected at parse time rather than failing per request.

Token Expired

Symptoms: “token expired” error.

Solution:

1. Request a new token from your identity provider
2. Check system clock synchronization:

   timedatectl status
   # Sync if needed
   sudo timedatectl set-ntp true

Wrong Algorithm

Symptoms: “unexpected signing method” error.

Solution:

Specify the allowed algorithms with algorithms (defaults to RS256 and ES256):

auth:
  providers:
    - name: jwt
      type: jwt
      enabled: true
      priority: 1
      config:
        jwks_url: "https://auth.example.com/.well-known/jwks.json"
        algorithms:
          - RS256
          - ES256

Adding an HS* entry to the list also requires hmac_secret in the same block.

---

API Key Authentication Issues

Invalid API Key

Symptoms: “invalid API key” error.

Solution:

1. Verify the key hash is correct:

   # Generate hash for your API key
   python3 -c "import bcrypt; print(bcrypt.hashpw(b'your-api-key', bcrypt.gensalt()).decode())"

2. Update configuration:

   auth:
     providers:
       - name: apikey
         type: apikey
         enabled: true
         priority: 1
         config:
           header_name: "X-API-Key"
           keys:
             - name: "service-a"
               key_hash: "$2a$10$correcthash..."

   keys is required and must contain at least one entry; every entry needs a name plus either key_hash or key_plain.

Wrong Header Name

Symptoms: API key not being recognized.

Solution:

Ensure the header name the client sends matches header_name (default X-API-Key):

auth:
  providers:
    - name: apikey
      type: apikey
      enabled: true
      priority: 1
      config:
        header_name: "X-API-Key"  # Client must use this exact header
        keys:
          - name: "service-a"
            key_hash: "$2a$10$correcthash..."

The setting is header_name, not header — an unrecognised key in a provider’s config block is silently ignored, so a misspelled name leaves the default X-API-Key in force.

# Test with correct header
curl -H "X-API-Key: your-api-key" http://localhost:7080/...

---

TOTP/HOTP Issues

Invalid Code

Symptoms: “invalid TOTP code” error.

Diagnosis:

1. Check time synchronization:

   timedatectl status

2. Verify secret is correctly configured

Solution:

1. Sync system time:

   sudo timedatectl set-ntp true

2. Verify secret format (must be Base32) and shape — secrets is a list of entries, not a username-to-secret mapping:

   auth:
     providers:
       - name: totp
         type: totp
         enabled: true
         priority: 1
         config:
           issuer: "Bifrost Proxy"
           secrets:
             - username: user1
               secret: "JBSWY3DPEHPK3PXP"  # Base32 encoded
               groups: ["users"]

   A map-shaped secrets: block is rejected at startup with totp config: secrets must be an array. Secrets can also be kept out of the main config with secrets_file, which holds the same list under a top-level secrets: key.

Time Drift

Symptoms: Codes work sometimes but not always.

Solution:

Ensure NTP is configured on all systems:

# Check NTP status
timedatectl show --property=NTPSynchronized

# Enable NTP
sudo timedatectl set-ntp true

---

mTLS Certificate Issues

Certificate Not Trusted

Symptoms: “certificate signed by unknown authority” error.

Solution:

Specify the CA certificate, either by path or inline:

auth:
  providers:
    - name: mtls
      type: mtls
      enabled: true
      priority: 1
      config:
        ca_cert_file: "/path/to/ca.crt"
        require_client_cert: true

One of ca_cert_file or ca_cert_pem (an inline PEM bundle) is required; the provider is rejected at startup if neither is present.

Certificate Expired

Symptoms: “certificate has expired” error.

Diagnosis:

# Check certificate expiration
openssl x509 -in client.crt -noout -dates

Solution:

Renew the client certificate.

Subject Not Allowed

Symptoms: subject not allowed: CN=... error.

Cause: The client certificate’s subject DN does not match any entry in allowed_subjects.

Solution:

allowed_subjects holds regular expressions matched against the full subject DN — there is no separate CN list and no shell-style wildcards. Restrict by CN with an anchored pattern:

auth:
  providers:
    - name: mtls
      type: mtls
      enabled: true
      priority: 1
      config:
        ca_cert_file: "/path/to/ca.crt"
        allowed_subjects:
          - "^CN=client1\\.example\\.com,.*$"
          - "^CN=[^,]+\\.internal\\.example\\.com,.*$"

Print the exact subject DN of a certificate before writing a pattern:

openssl x509 -in client.crt -noout -subject

Issuers can be restricted the same way with allowed_issuers. To pick the username and groups out of the certificate, use subject_mapping with username_field, groups_field, and email_field.

---

MFA Wrapper Issues

Primary Authentication Failed

Symptoms: First factor (password) fails.

Solution:

Check the primary authenticator inside the wrapper. Both primary: and secondary: blocks are required — the wrapper fails to start if either is missing, so verify the primary factor in place:

auth:
  providers:
    - name: mfa
      type: mfa_wrapper
      enabled: true
      priority: 1
      config:
        primary:
          mode: native
          config:
            users:
              - username: admin
                password_hash: "$2a$12$..."
        secondary:
          mode: totp
          config:
            secrets:
              - username: admin
                secret: "JBSWY3DPEHPK3PXP"
        mfa_required: always
        password_format: separated
        separator: ":"

Each block names a plugin type in mode: and passes that plugin’s own settings in config:. Referring to providers by name instead of inlining them is not supported.

OTP Not Provided

Symptoms: “OTP required” error.

Solution:

Provide OTP in one of these ways:

1. Via header:

   curl -x http://user:password@localhost:7080 \
     -H "X-OTP: 123456" \
     https://example.com

2. Appended to password:

   # With password_format "separated" and separator ":"
   curl -x http://user:password:123456@localhost:7080 https://example.com

   The setting is separator (otp_separator is still accepted as an alias for backwards compatibility).

---

Diagnostic Commands

# Test authentication
curl -x http://user:password@localhost:7080 https://httpbin.org/ip -v

# Check auth errors in logs
journalctl -u bifrost-server | grep -E "auth|401|403|407"

# Validate configuration
bifrost-server validate -c config.yaml

# Test LDAP connectivity
ldapsearch -x -H ldap://ldap.example.com -D "cn=admin,dc=example,dc=com" -W

# Check JWT token
echo "token" | cut -d. -f2 | base64 -d | jq

# Verify bcrypt hash
python3 -c "import bcrypt; print(bcrypt.checkpw(b'password', b'\$2a\$12\$hash'))"