Authentication does not work with JupyterHub 4.1.2 #1842

trungleduc · 2024-04-09T13:56:16Z

Bug description

I'm using the local-binder-local-hub setup with auth enabled (c.BinderHub.auth_enabled = True). This setup works well up to JupyterHub 4.0.2.

Starting from 4.1.2, it does not work anymore with the following errors

[I 2024-04-09 15:50:00.086 JupyterHub log:192] 200 POST /hub/api/oauth2/token (binder@10.255.255.254) 33.50ms
[D 240409 15:50:00 auth:527] HubAuth cache miss: token:76615381b56649daa983eeb0db8d467c:88eec0b396b0b2a4c89d43d6edcd220c8146120210ba068ec8f6c9231cc00ea6
[I 2024-04-09 15:50:00.101 JupyterHub log:192] 200 GET /hub/api/user (alice@10.255.255.254) 13.31ms
[D 240409 15:50:00 auth:538] Received request from Hub user {'groups': [], 'kind': 'user', 'admin': False, 'name': 'alice', 'session_id': '76615381b56649daa983eeb0db8d467c', 'scopes': ['access:services!service=binder', 'read:users:groups!user=alice', 'read:users:name!user=alice']}
[I 240409 15:50:00 auth:1527] Logged-in user {'groups': [], 'kind': 'user', 'admin': False, 'name': 'alice', 'session_id': '76615381b56649daa983eeb0db8d467c', 'scopes': ['access:services!service=binder', 'read:users:groups!user=alice', 'read:users:name!user=alice']}
[D 240409 15:50:00 auth:1195] Setting oauth cookie for 127.0.0.1: service-binderhub, {'path': '/services/binder/', 'httponly': True}
[I 240409 15:50:00 _xsrf_utils:125] Setting new xsrf cookie for b'76615381b56649daa983eeb0db8d467c:88eec0b396b0b2a4c89d43d6edcd220c8146120210ba068ec8f6c9231cc00ea6' {'path': '/services/binder/'}
[D 240409 15:50:00 auth:977] Not accepting cookie auth on GET /services/binder/oauth_callback: '_xsrf' argument missing from GET
[D 240409 15:50:00 auth:781] No user identified

and in the browser:

GET http://127.0.0.1:8000/services/binder/oauth_callback?code=RjcOLjkaqwPffgVJsEN4h6RqWOX1RF&state=eyJ1dWlkIjogIjRm...QUQifQ net::ERR_TOO_MANY_REDIRECTS

How to reproduce

Install JupyterHub 4.1.2
Set up binderhub with local-binder-local-hub
Switch c.JupyterHub.authenticator_class to DummyAuthenticator
Add c.BinderHub.auth_enabled = True to binderhub_config.py
Launch any repository

Expected behaviour

The builder process should start

Actual behaviour

Your personal set up

binderhub at main
jupyterhub 4.1.2

Full environment

# paste output of `pip freeze` or `conda list` here

Configuration

# jupyterhub_config.py

Logs

The text was updated successfully, but these errors were encountered:

welcome · 2024-04-09T13:56:19Z

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! 🤗

If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively.

You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! 👋

Welcome to the Jupyter community! 🎉

trungleduc · 2024-04-09T13:59:16Z

It might be related to jupyterhub/jupyterhub#4761. @minrk Is there any config I need to add to make it work?

manics · 2024-04-09T17:20:35Z

I've tried this with JupyterHub 4.1.5. I also get a failure, though it's different to yours:

[I 2024-04-09 18:19:02.677 JupyterHub log:192] 302 GET / -> /hub/ (@127.0.0.1) 1.06ms
[I 2024-04-09 18:19:02.689 JupyterHub log:192] 302 GET /hub/ -> /services/binder/ (as@127.0.0.1) 4.10ms
[D 240409 18:19:02 auth:838] No user identified
[D 240409 18:19:02 auth:1395] Redirecting to login url: /hub/api/oauth2/authorize?client_id=service-binder&redirect_uri=%2Fservices%2Fbinder%2Foauth_callback&response_type=code
[W 240409 18:19:02 auth:1160] Detected unused OAuth state cookies
[I 240409 18:19:02 log:135] 302 GET /services/binder/ -> /hub/api/oauth2/authorize?client_id=service-binder&redirect_uri=%2Fservices%2Fbinder%2Foauth_callback&response_type=code&state=[secret] (@127.0.0.1) 1.59ms
[D 2024-04-09 18:19:02.709 JupyterHub provider:421] Validating client id service-binder
[D 2024-04-09 18:19:02.710 oauthlib.oauth2.rfc6749.grant_types.authorization_code authorization_code:362] Validating redirection uri /services/binder/oauth_callback for client service-binder.
[D 2024-04-09 18:19:02.710 oauthlib.oauth2.rfc6749.grant_types.base base:231] Using provided redirect_uri /services/binder/oauth_callback
[D 2024-04-09 18:19:02.710 JupyterHub provider:497] validate_redirect_uri: client_id=service-binder, redirect_uri=/services/binder/oauth_callback
[D 2024-04-09 18:19:02.711 oauthlib.oauth2.rfc6749.grant_types.base base:172] Validating access to scopes ['read:users:groups!user', 'read:users:name!user', 'access:services!service=binder'] for client 'service-binder' (<OAuthClient(identifier='service-binder')>).
[D 2024-04-09 18:19:02.711 JupyterHub provider:624] Allowing request for scope(s) for service-binder:  read:users:groups!user,read:users:name!user,access:services!service=binder
[E 2024-04-09 18:19:02.712 JupyterHub auth:289] User <User(as 0/1 running)> not allowed to access JupyterHub service binder
[W 2024-04-09 18:19:02.712 JupyterHub web:1873] 403 GET /hub/api/oauth2/authorize?client_id=service-binder&redirect_uri=%2Fservices%2Fbinder%2Foauth_callback&response_type=code&state=<state> (127.0.0.1): You do not have permission to access JupyterHub service binder
[D 2024-04-09 18:19:02.712 JupyterHub base:1513] No template for 403
[W 2024-04-09 18:19:02.713 JupyterHub log:192] 403 GET /hub/api/oauth2/authorize?client_id=service-binder&redirect_uri=%2Fservices%2Fbinder%2Foauth_callback&response_type=code&state=[secret] (as@127.0.0.1) 8.15ms

minrk · 2024-04-10T11:00:15Z

There are fixes since 4.1.2 that are probably relevant, specifically jupyterhub/jupyterhub#4759 in 4.1.4, I believe. I do not think jupyterhub/jupyterhub#4761 is related, since that was specifically a CORS request that is now correctly blocked.

The error @manics sees is correctly raising because the test user doesn't have permission to visit the service (access:services!service=binder).

trungleduc · 2024-04-10T11:22:08Z

Just tested with 4.1.5, I still have the same issue

dlamoris · 2024-05-02T21:13:23Z

I have an internal k8s deployment that have similar symptoms, adding some data point to hopefully help with debugging.

Using binderhub chart version 1.0.0-0.dev.git.3431.h8a8a59e with jupyterhub/k8s-binderhub image 1.0.0-0.dev.git.3422.h35a34dd, I get the same error "Failed to connect to event stream" when trying to launch a repo. Attached is the har file from chrome's dev tool network. You'll see from the Get call to https://cae-binder-uat.jpl.nasa.gov/services/binder/build/gh/dlam/conda/main it gets a return of 302 and starts an oauth loop that results in too many redirects. This is interesting because the jupyterhub is 4.0.2 here (it's using quay.io/jupyterhub/k8s-hub:3.2.1 which should have 4.0.2)

bad-binder.har.zip

Our last known working version of binderhub image is 1.0.0-0.dev.git.2809.h97a8ce1, and if I use that image with everything else in the chart staying the same, the call and launch works and instead of 302 returns 200. I have not tried other images in between that and latest. (although 1.0.0-0.dev.git.2816.h5db2f98 results in a 500 when going to binder page)

sample chart value of the deployment, using chart 1.0.0-0.dev.git.3431.h8a8a59e:

service:
  type: ClusterIP
image:
  name: jupyterhub/k8s-binderhub
  tag: 1.0.0-0.dev.git.2809.h97a8ce1
registry:
  password: <replaced>
  url: <url>
  username: <username>
imageBuilderType: dind
ingress:
  enabled: false # using jupyterhub's ingress
config:
  BinderHub:
    debug: true
    base_url: "/services/binder/" # from jupyterhub proxy
    use_registry: true
    auth_enabled: true
    hub_url: "https://cae-binder-uat.jpl.nasa.gov/" # this refers to jupyterhub
    image_prefix: <prefix>
    build_image: "quay.io/jupyterhub/repo2docker:2023.06.0"
  DockerRegistry:
    url: "url"
    auth_config_url: "url"
    token_url: "url"
  KubernetesBuildExecutor:
    build_image: "quay.io/jupyterhub/repo2docker:2023.06.0" 
  GitHubRepoProvider:
    access_token: <replaced>
    api_base_path: https://{hostname}/api/v3
    hostname: github.jpl.nasa.gov
jupyterhub:
  proxy:
    https:
      enabled: false
    secretToken: <secret>
    service:
      type: ClusterIP
  debug:
    enabled: true
  hub:
    config:
      BinderSpawner:
        auth_enabled: true
      Authenticator:
        admin_users:
          - admin
        enable_auth_state: true
      JupyterHub:
        admin_access: true
        authenticator_class: generic-oauth
      GenericOAuthenticator:
        allow_all: true
        authorize_url: >-
          <url>
        client_id: cae-binder-uat
        client_secret: <secret>
        login_service: CAE SSO
        oauth_callback_url: https://cae-binder-uat.jpl.nasa.gov/hub/oauth_callback
        scope:
          - openid
        token_url: >-
          <url>
        userdata_params:
          state: state
        userdata_url: >-
          <url>
        username_key: preferred_username

    allowNamedServers: true
    namedServerLimitPerUser: 10
    redirectToServer: false
    services:
      binder:
        oauth_no_confirm: true
        url: "http://binder.cae-binder-uat.svc.cluster.local" # jupyterhub will add /services/binder/ in its proxy to this url
        oauth_redirect_uri: "https://cae-binder-uat.jpl.nasa.gov/services/binder/oauth_callback"
        oauth_client_id: "service-cae-binder-uat-oauth-client-uat"
    loadRoles:
      binder:
        scopes:
          - servers
          - admin:users
        services:
          - binder
      user:
        scopes:
          - self
          - access:services
    extraConfig:
      01-hub_extra: |
        c.JupyterHub.default_url = '/services/binder/'
      02-timeout: |
        c.Spawner.http_timeout = 1800
  singleuser:
    cmd: jupyterhub-singleuser
    defaultUrl: "/lab"
  ingress:
    enabled: true
    hosts:
      - cae-binder-uat.jpl.nasa.gov
    pathSuffix: null
    pathType: Prefix
    tls:
      - hosts:
          - cae-binder-uat.jpl.nasa.gov
        secretName: cae-binder-uat

minrk · 2024-05-03T12:25:41Z

#1847 should fix this.

trungleduc added the bug label Apr 9, 2024

This was referenced Apr 9, 2024

Z2JH 3.3.7 manics/binderhub#9

Closed

Z2JH 3.3.7 #1843

Merged

minrk mentioned this issue May 3, 2024

handle api tokens and xsrf #1847

Merged

yuvipanda closed this as completed in #1847 May 14, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Authentication does not work with JupyterHub 4.1.2 #1842

Authentication does not work with JupyterHub 4.1.2 #1842

trungleduc commented Apr 9, 2024 •

edited

welcome bot commented Apr 9, 2024

trungleduc commented Apr 9, 2024

manics commented Apr 9, 2024

minrk commented Apr 10, 2024

trungleduc commented Apr 10, 2024

dlamoris commented May 2, 2024 •

edited

minrk commented May 3, 2024

Authentication does not work with JupyterHub 4.1.2 #1842

Authentication does not work with JupyterHub 4.1.2 #1842

Comments

trungleduc commented Apr 9, 2024 • edited

Bug description

How to reproduce

Expected behaviour

Actual behaviour

Your personal set up

welcome bot commented Apr 9, 2024

trungleduc commented Apr 9, 2024

manics commented Apr 9, 2024

minrk commented Apr 10, 2024

trungleduc commented Apr 10, 2024

dlamoris commented May 2, 2024 • edited

minrk commented May 3, 2024

trungleduc commented Apr 9, 2024 •

edited

dlamoris commented May 2, 2024 •

edited