JWT Token Introspection Request Fails through NGINX Gateway

[archmangler]

I'm trying to set up token inspection with keycloak using the instructions here:

https://github.com/nginxinc/NGINX-Demos/tree/master/oauth2-token-introspection-oss

Using NGINX as a gateway to do the token introspection fails with 403 forbidden, however if I send the token introspection request directly to keycloak server it is successful:

Note, I follow two steps here:

a) Request JWT bearer token from keycloak via NGINX gateway.
b) Make an API request via the NGINX api gateway which uses token introspection to authorize the request.

(Included NGINX configuration at the bottom of this post)

1. Healthy/Successful Introspection Request (directly against introspection endpoint):

curl -k -v \
     -X POST \
     -u "$KC_CLIENT:$KC_CLIENT_SECRET" \
     -d "token=$BEARER" \
     "https://$KC_SERVER:8443/realms/$KC_REALM/protocol/openid-connect/token/introspect"\
     | jq .

• Result of curl directly to Keycloak introspection endpoint:

* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [41 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [1083 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: O=mkcert development certificate; OU=root@beta.engeneon.com
*  start date: May 29 10:24:48 2023 GMT
*  expire date: Aug 29 10:24:48 2025 GMT
*  issuer: O=mkcert development CA; OU=root@beta.engeneon.com; CN=mkcert root@beta.engeneon.com
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Server auth using Basic with user 'WPPI.UKT'
* Using Stream ID: 1 (easy handle 0x55b331ef2db0)
} [5 bytes data]

> POST /realms/hkjc-api-dev/protocol/openid-connect/token/introspect HTTP/2

> Host: 10.0.0.5:8443
> authorization: Basic V1BQSS5VS1Q6VU5RaE9rYml4bDEzTVRwU2ZvUk5KaUFXanVNOHY2cU0=
> user-agent: curl/7.68.0
> accept: */*
> content-length: 1203
> content-type: application/x-www-form-urlencoded

> 
} [5 bytes data]
* We are completely uploaded and fine
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [50 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
} [5 bytes data]
< HTTP/2 200 
< referrer-policy: no-referrer
< x-frame-options: SAMEORIGIN
< strict-transport-security: max-age=31536000; includeSubDomains
< x-content-type-options: nosniff
< x-xss-protection: 1; mode=block
< content-type: application/json
< content-length: 839
< 
{ [5 bytes data]
^M100  2042  100   839  100  1203  25424  36454 --:--:-- --:--:-- --:--:-- 68066
* Connection #0 to host 10.0.0.5 left intact
{
  "exp": 1685513603,
  "iat": 1685513303,
  "jti": "8653cd7a-d205-4626-93e6-5cb3998ead4e",
  "iss": "https://beta.engeneon.com:8443/realms/hkjc-api-dev",
  "aud": [
    "WPPI.UKT",
    "account"
  ],
  "sub": "c391492d-23d9-4d9f-b99d-d3327299b754",
  "typ": "Bearer",
  "azp": "WPPI.UKT",
  "session_state": "8b638382-e431-4f30-a57f-014d26623c08",
  "preferred_username": "service-account-wppi.ukt",
  "email_verified": false,
  "acr": "1",
  "allowed-origins": [
    "/*"
  ],
  "realm_access": {
    "roles": [
      "default-roles-hkjc-api-dev",
      "offline_access",
      "uma_authorization"
    ]
  },
  "resource_access": {
    "WPPI.UKT": {
      "roles": [
        "uma_protection"
      ]
    },
    "account": {
      "roles": [
        "manage-account",
        "manage-account-links",
        "view-profile"
      ]
    }
  },
  "scope": "profile email txn_gp9",
  "sid": "8b638382-e431-4f30-a57f-014d26623c08",
  "clientHost": "10.0.0.4",
  "clientAddress": "10.0.0.4",
  "client_id": "WPPI.UKT",
  "username": "service-account-wppi.ukt",
  "active": true
}

2. Failing introspection request (through NGINX API gateway)

• Curl script:

#!/bin/bash

KC_CLIENT="WPPI.UKT"
KC_CLIENT_SECRET="UNQhOkbixl13MTpSfoRNJiAWjuM8v6qM"
KC_SERVER="10.0.0.5"
KC_CONTEXT="auth"
KC_REALM="hkjc-api-dev"

BEARER=$(curl -k -L -X POST 'https://alpha/auth/realms/hkjc-api-dev/protocol/openid-connect/token'    -H 'Content-Type: application/x-www-form-urlencoded'    --data-urlencode 'client_id=WPPI.UKT'    --data-urlencode 'grant_type=client_credentials'    --data-urlencode 'client_secret=UNQhOkbixl13MTpSfoRNJiAWjuM8v6qM'    --data-urlencode 'scope=txn_gp9'| jq -r  | jq -r '.access_token')

curl -k -v \
     -X POST \
     -u "$KC_CLIENT:$KC_CLIENT_SECRET" \
     -d "token=$BEARER" \
     "https://alpha/" | jq -r .

Curl-side debug:

* TCP_NODELAY set
* Connected to alpha (10.0.0.4) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [19 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [942 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=SG; ST=Changi; L=Singapore; O=Engeneon; OU=Division; CN=Alpha; emailAddress=traiano@gmail.com
*  start date: May 18 16:42:48 2023 GMT
*  expire date: May 17 16:42:48 2024 GMT
*  issuer: C=SG; ST=Changi; L=Singapore; O=Engeneon; OU=Division; CN=Alpha; emailAddress=traiano@gmail.com
*  SSL certificate verify result: self signed certificate (18), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Server auth using Basic with user 'WPPI.UKT'
* Using Stream ID: 1 (easy handle 0x55e19f784db0)
} [5 bytes data]

* Server auth using Basic with user 'WPPI.UKT'
* Using Stream ID: 1 (easy handle 0x55e19f784db0)
} [5 bytes data]
> POST / HTTP/2
> Host: alpha
> authorization: Basic V1BQSS5VS1Q6VU5RaE9rYml4bDEzTVRwU2ZvUk5KaUFXanVNOHY2cU0=
> user-agent: curl/7.68.0
> accept: */*
> content-length: 1203
> content-type: application/x-www-form-urlencoded
> 
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [265 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [249 bytes data]
* old SSL session ID is stale, removing
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
} [5 bytes data]
* We are completely uploaded and fine
{ [5 bytes data]

< HTTP/2 403 
< server: nginx/1.24.0
< date: Wed, 31 May 2023 06:14:42 GMT
< content-type: text/html
< content-length: 153
< 
{ [153 bytes data]
^M100  1356  100   153  100  1203   4371  34371 --:--:-- --:--:-- --:--:-- 38742
* Connection #0 to host alpha left intact
<html>
<head><title>403 Forbidden</title></head>
<body>
<center><h1>403 Forbidden</h1></center>
<hr><center>nginx/1.24.0</center>
</body>
</html>

• Keycloak server logs indicate the client wan't found clientId=null, userId=null, ipAddress=10.0.0.4, error=client_not_found, though in then nginx logs it shows the client credentials are correctly converted to base64:

2023-05-31 06:23:02,540 WARN  [org.keycloak.events] (executor-thread-21) type=INTROSPECT_TOKEN_ERROR, realmId=84d8e944-8143-4cc7-8dcd-128e2ec0ebfb, clientId=null, userId=null, ipAddress=10.0.0.4, error=client_not_found
2023-05-31 06:23:02,540 WARN  [org.keycloak.events] (executor-thread-21) type=INTROSPECT_TOKEN_ERROR, realmId=84d8e944-8143-4cc7-8dcd-128e2ec0ebfb, clientId=null, userId=null, ipAddress=10.0.0.4, error=invalid_request, detail='Authentication failed.'

NGINX API gateway logs:

• Got bearer token from KC via Nginx api gw:

==> /var/log/nginx/access.log <==
10.0.0.6 - - [31/May/2023:06:26:37 +0000] "POST /auth/realms/hkjc-api-dev/protocol/openid-connect/token HTTP/2.0" 200 2109 "-" "curl/7.68.0" "-"

• Token introspection fails:

==> /var/log/nginx/error.log <==
2023/05/31 06:26:37 [info] 10712#10712: *30 js: DEBUG: BEFORE: OAuth sending introspection request with token: Basic V1BQSS5VS1Q6VU5RaE9rYml4bDEzTVRwU2ZvUk5KaUFXanVNOHY2cU0=

??? -> 2023/05/31 06:26:37 [info] 10712#10712: *30 js: OAuth Got AuthHeader:  Basic my-client-id:my-client-secret

2023/05/31 06:26:37 [info] 10712#10712: *30 js: DEBUG: AFTER: OAuth sending introspection request with token: Basic V1BQSS5VS1Q6VU5RaE9rYml4bDEzTVRwU2ZvUk5KaUFXanVNOHY2cU0=
2023/05/31 06:26:37 [info] 10712#10712: *30 js: OAuth sending introspection request with token: Basic V1BQSS5VS1Q6VU5RaE9rYml4bDEzTVRwU2ZvUk5KaUFXanVNOHY2cU0=
2023/05/31 06:26:37 [error] 10712#10712: *30 js: OAuth unexpected response from authorization server (HTTP 401). undefined
2023/05/31 06:26:37 [info] 10712#10712: *30 js: OAuth token introspection response: {"error":"invalid_request","error_description":"Authentication failed."}
2023/05/31 06:26:37 [warn] 10712#10712: *30 js: OAuth token introspection found inactive token

• Final access log entry ("403")

==> /var/log/nginx/access.log <==
10.0.0.6 - WPPI.UKT [31/May/2023:06:26:37 +0000] "POST / HTTP/2.0" 403 153 "-" "curl/7.68.0" "-"

Nginx.conf:

js_import scripts/oauth2.js;

map $http_authorization $access_token {
    "~*^Bearer (.*)$" $1;
    default $http_authorization;
}

#OAuth 2.0 Token Introspection configuration
#proxy_cache_path /var/cache/nginx/tokens levels=1 keys_zone=token_responses:1m max_size=10m;
#resolver 8.8.8.8;                  # For DNS lookup of OAuth server
subrequest_output_buffer_size 16k; # To fit a complete response from OAuth server

server {

    listen              443 ssl http2;

    server_name         alpha.engeneon.com;
    ssl_certificate     alpha.engeneon.com.crt;
    ssl_certificate_key alpha.engeneon.com.key;
    ssl_protocols       TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;
    ssl_ciphers         HIGH:!aNULL:!MD5;

    #set $access_token $http_apikey; # Where to find the token. Remove when using Authorization header
    #e.g "https://$KC_SERVER:8443/realms/$KC_REALM/protocol/openid-connect/token/introspect"

    set $oauth_token_endpoint     "https://10.0.0.5:8443/realms/hkjc-api-dev/protocol/openid-connect/token/introspect";
    set $oauth_token_hint         "access_token"; # E.g. access_token, refresh_token
    set $oauth_client_id          "my-client-id"; # Will use HTTP Basic authentication unless empty
    set $oauth_client_secret      "my-client-secret"; # If id is empty this will be used as a bearer token

    proxy_set_header X-Forwarded-For $proxy_protocol_addr;          # To forward the original client's IP address 
    proxy_set_header X-Forwarded-Proto $scheme;                     # to forward the  original protocol (HTTP or HTTPS)

    #Client Step #1: First get a JWT
    location /auth/ {
      proxy_pass https://10.0.0.5:8443/;
    }

    location / {
        auth_request /_oauth2_token_introspection;

        # Any member of the token introspection response is available as $sent_http_token_member
        #auth_request_set $username $sent_http_token_username;
        #proxy_set_header X-Username $username;

        #pass through to API endpoint once the JWT has been authorized by introspection
        proxy_pass http://10.0.0.7;
    }

    location = /_oauth2_token_introspection {
        # This location implements an auth_request server that uses the JavaScript
        # module to perform the token introspection request.
        internal;
        js_content oauth2.introspectAccessToken;
    }

    location = /_oauth2_send_introspection_request {
        # This location is called by introspectAccessToken(). We use the proxy_
        # directives to construct an OAuth 2.0 token introspection request, as per:
        #  https://tools.ietf.org/html/rfc7662#section-2
        internal;
        gunzip on; # Decompress if necessary

        proxy_method      POST;
        proxy_set_header  Authorization $arg_authorization;
        proxy_set_header  Content-Type "application/x-www-form-urlencoded";
        proxy_set_body    "token=$arg_token&token_hint=$oauth_token_hint";
        proxy_pass        $oauth_token_endpoint;

    }

}