Session refresh issue with Jumpcloud IdP

[gmsoft-tuxicoman]

What happened?

We've used Jumpcloud as an identity provider. We can authenticate correctly and see all the claims in verify.
We are currently trying to provide access to jira and splunk. Both of those fail after a while. Looking at the HTTP logs, after a while, about 30 minutes, the API calls are redirected to the authenticate service URL to login again.

What did you expect to happen?

The session should be alive for more than 30 minutes.

How'd it happen?

Browse splunk.mycompany.com.
Use splunk for a while.
After ~30 mins API calls to splunk are redirected to the authenticate URL. The user facing error message is "Disconnected from Splunk server."

What's your environment like?

• Pomerium version 0.24
• Running on debian 11 using the debian package.

What's your config.yaml?

address: :8443
authenticate_service_url: https://authn-pomerium.mycompany.com
authorize_service_url: https://authz-pomerium.mycompany.com
certificate_file: /etc/ssl/private/mycompany.com/mycompany.com.crt
certificate_key_file: /etc/ssl/private/mycompany.com/mycompany.com.key
cookie_secret: REDACTED
idp_provider: oidc
idp_provider_url: https://oauth.id.jumpcloud.com/
idp_scopes:
- openid
- email
- profile
- groups
- offline_access
idp_request_params:
  response_type: code
  max_age: 86400
  prompt: consent

routes:
- allow_any_authenticated_user: true
  from: https://jira.mycompany.com
  idp_client_id: REDACTED
  idp_client_secret: REDACTED
  to: https://jira.mycompany.com
- allow_any_authenticated_user: true
  from: https://confluence.mycompany.com
  idp_client_id: REDACTED
  idp_client_secret: REDACTED
  to: https://confluence.mycompany.com
- allow_any_authenticated_user: true
  from: https://splunk.mycompany.com
  idp_client_id: REDACTED
  idp_client_secret: REDACTED
  to: https://splunk.mycompany.com

At first I tried without the idp_request_params but the problem is the same.

What did you see in the logs?

Nov 21 10:51:53 slc-sec-pomerium1 pomerium[1213888]: {"level":"info","config_file_source":"/etc/pomerium/config.yaml","bootstrap":true,"service":"identity_manager","user_id":"5f15c0b56826ac55192595d5","session_id":"0f8e981a-a304-4fd7-a5ac-ed8c2c0edd72","time":"2023-11-21T10:51:53Z","message":"refreshing session"}
Nov 21 10:51:53 slc-sec-pomerium1 pomerium[1213888]: {"level":"error","config_file_source":"/etc/pomerium/config.yaml","bootstrap":true,"service":"identity_manager","error":"identity/oidc: refresh failed: oauth2: \"invalid_request\" \"The request is missing a required parameter, includes an invalid parameter value, includes a parameter more than once, or is otherwise malformed. Client credentials missing or malformed in both HTTP Authorization header and HTTP POST body.\"","user_id":"5f15c0b56826ac55192595d5","session_id":"0f8e981a-a304-4fd7-a5ac-ed8c2c0edd72","time":"2023-11-21T10:51:53Z","message":"failed to refresh oauth2 token, deleting session"}

Additional context

I couldn't find a way to get logs from the OIDC queries made to the IdP. That would help me understand the issue.

[wasaga]

One way to investigate is to configure pomerium to use Postgres databroker backend.

Then look for the contents of pomerium.records table that have "Session" in type column.

Next look for any errors in pomerium logs from identity_manager. There's a background process that continuously refreshes sessions with the IdP

[gmsoft-tuxicoman]

I've done that but it doesn't provide more info than what I already had and provided above.

The request to the IdP is failing with this:

The request is missing a required parameter, includes an invalid parameter value, includes a parameter more than once, or is otherwise malformed. Client credentials missing or malformed in both HTTP Authorization header and HTTP POST body.

Unfortunately I can't see what requests pomerium is issuing to Jumpcloud. How can I troubleshoot that ?

[desimone]

@gmsoft-tuxicoman

A few quick thoughts looking at your config:

• Indentation seems off in several places
• Try setting a global client_id/secret in the top level config to see if that works

Otherwise, If you are able to build from source and add additional log messages to the refresh call would be the best way to troubleshoot why the interaction between Pomerium's session refresh & jumpcloud is failing conclusively.

[gmsoft-tuxicoman]

@desimone I deleted my previous post. It seems that the issue keeps occurring.
For some reason the refresh session query provides empty client id/secret when refreshing the session token.

I used mitmproxy to sniff the traffic between pomerium and Jumpcloud. Pomerium send Authorization: Basic Og==. This is just : for the credential.

Shouldn't pomerium provide some kind of authentication ?
The only 2 parameters in the query are : grant_type: refresh_token, refresh_token: <old refresh_token>.

[gmsoft-tuxicoman]

Ok it looks like I was reviewing old logs. Adding the client id and secret in the top does solve the issue.
When I review the queries made by pomerium for the refresh_token, I definitely see the client_id and client_secret now.

[gmsoft-tuxicoman]

Anything I can do for this one ? Thanks

[kenjenkins]

@gmsoft-tuxicoman, is using a single global IdP client ID & secret a viable workaround for your deployment?

I think the underlying issue here is that session refresh will use only the global IdP settings, rather than any per-route IdP client ID / secret.

Session refresh is handled by a component called the identity manager. It looks like the identity manager is configured with a single IdP authenticator, which is created based on the main IdP configuration settings.

[gmsoft-tuxicoman]

@kenjenkins Yes I have configured a single idp entry for now and doing the authorization in pomerium. That workaround is working.