You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I'm trying to run ml-hub with ml-workspace but having problems starting the spawner. The hub is up and running, logging in users (ldap) is working fine but it fails when i start the spawner.
The setup is pretty much standard except that
i'm using podman instead of docker (tried both running as root and rootless)
overriding the jupyterhub_config.py to be able to modify the function that renames the container (not working in podman)
Technical
Log output
Starting ML Hub
No certificate was provided for SSL/HTTPS.
Generate self-signed certificate for SSL/HTTPS.
Start SSH Daemon service
Start JupyterHub
Warning: If you want to use the SSH feature, you have to start the hub with ssl enabled.
Start nginx
[I 2021-11-02 06:10:54.127 JupyterHub app:2120] Using Authenticator: ldapauthenticator.ldapauthenticator.LDAPAuth enticator-1.3.2
[I 2021-11-02 06:10:54.128 JupyterHub app:2120] Using Spawner: mlhubspawner.mlhubspawner.MLHubDockerSpawner
[I 2021-11-02 06:10:54.136 JupyterHub app:1257] Loading cookie_secret from /data/jupyterhub_cookie_secret
[D 2021-11-02 06:10:54.148 JupyterHub app:1424] Connecting to db: sqlite:////data/jupyterhub.sqlite
[D 2021-11-02 06:10:54.184 JupyterHub orm:749] database schema version found: 4dc2d5a8c53c
[I 2021-11-02 06:10:54.198 JupyterHub proxy:460] Generating new CONFIGPROXY_AUTH_TOKEN
[I 2021-11-02 06:10:54.253 JupyterHub app:1563] Not using whitelist. Any authenticated user will be allowed.
[D 2021-11-02 06:10:54.368 JupyterHub app:1910] Loading state for xxx001 from db
[D 2021-11-02 06:10:54.369 JupyterHub app:1910] Loading state for admin from db
[D 2021-11-02 06:10:54.369 JupyterHub app:1926] Loaded users:
xxx001 admin
admin admin
[I 2021-11-02 06:10:54.382 JupyterHub app:2337] Hub API listening on http://0.0.0.0:8081/hub/
[I 2021-11-02 06:10:54.383 JupyterHub app:2339] Private Hub API connect url http://188f0675ac42:8081/hub/
[W 2021-11-02 06:10:54.400 JupyterHub proxy:642] Running JupyterHub without SSL. I hope there is SSL termination happening somewhere else...
[I 2021-11-02 06:10:54.400 JupyterHub proxy:645] Starting proxy @ http://:8000
[D 2021-11-02 06:10:54.400 JupyterHub proxy:646] Proxy cmd: ['configurable-http-proxy', '--ip', '', '--port', '80 00', '--api-ip', '127.0.0.1', '--api-port', '8001', '--error-target', 'http://188f0675ac42:8081/hub/error']
[D 2021-11-02 06:10:54.426 JupyterHub proxy:561] Writing proxy pid file: jupyterhub-proxy.pid
06:10:54.850 [ConfigProxy] info: Proxying http://*:8000 to (no default)
06:10:54.852 [ConfigProxy] info: Proxy API at http://127.0.0.1:8001/api/routes
[D 2021-11-02 06:10:55.331 JupyterHub proxy:681] Proxy started and appears to be up
[I 2021-11-02 06:10:55.331 JupyterHub app:2362] Starting managed service cleanup-service at http://127.0.0.1:9000
[I 2021-11-02 06:10:55.332 JupyterHub service:316] Starting service 'cleanup-service': ['/usr/bin/python3', '/res ources/cleanup-service.py']
[I 2021-11-02 06:10:55.339 JupyterHub service:121] Spawning /usr/bin/python3 /resources/cleanup-service.py
[D 2021-11-02 06:10:55.351 JupyterHub spawner:1084] Polling subprocess every 30s
WARNING:tornado.access:404 GET /services/cleanup-service/ (127.0.0.1) 0.55ms
[D 2021-11-02 06:10:57.353 JupyterHub utils:218] Server at http://127.0.0.1:9000/services/cleanup-service/ respon ded with 404
[D 2021-11-02 06:10:57.353 JupyterHub proxy:314] Fetching routes to check
[D 2021-11-02 06:10:57.354 JupyterHub proxy:765] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
[I 2021-11-02 06:10:57.361 JupyterHub proxy:319] Checking routes
[I 2021-11-02 06:10:57.361 JupyterHub proxy:399] Adding default route for Hub: / => http://188f0675ac42:8081
[W 2021-11-02 06:10:57.362 JupyterHub proxy:373] Adding missing route for cleanup-service (Server(url=http://127. 0.0.1:9000/services/cleanup-service/, bind_url=http://127.0.0.1:9000/services/cleanup-service/))
[D 2021-11-02 06:10:57.363 JupyterHub proxy:765] Proxy: Fetching POST http://127.0.0.1:8001/api/routes/
[I 2021-11-02 06:10:57.364 JupyterHub proxy:242] Adding service cleanup-service to proxy /services/cleanup-servic e/ => http://127.0.0.1:9000
06:10:57.372 [ConfigProxy] info: 200 GET /api/routes
[D 2021-11-02 06:10:57.373 JupyterHub proxy:765] Proxy: Fetching POST http://127.0.0.1:8001/api/routes/services/c leanup-service
06:10:57.375 [ConfigProxy] info: Adding route / -> http://188f0675ac42:8081
06:10:57.376 [ConfigProxy] info: Route added / -> http://188f0675ac42:8081
06:10:57.376 [ConfigProxy] info: 201 POST /api/routes/
06:10:57.377 [ConfigProxy] info: Adding route /services/cleanup-service -> http://127.0.0.1:9000
06:10:57.377 [ConfigProxy] info: Route added /services/cleanup-service -> http://127.0.0.1:9000
06:10:57.377 [ConfigProxy] info: 201 POST /api/routes/services/cleanup-service
[I 2021-11-02 06:10:57.378 JupyterHub app:2422] JupyterHub is now running at http://:8000
[I 2021-11-02 06:11:16.425 JupyterHub log:174] 302 GET / -> /hub/ (@::ffff:10.162.2.18) 1.59ms
[D 2021-11-02 06:11:16.454 JupyterHub base:289] Refreshing auth for xxx001
[I 2021-11-02 06:11:16.455 JupyterHub log:174] 302 GET /hub/ -> /hub/home (xxx001@::ffff:10.162.2.18) 13.83ms
[D 2021-11-02 06:11:16.522 JupyterHub user:240] Creating <class 'mlhubspawner.mlhubspawner.MLHubDockerSpawner'> f or xxx001:
[I 2021-11-02 06:11:17.062 JupyterHub log:174] 200 GET /hub/home (xxx001@::ffff:10.162.2.18) 577.46ms
[D 2021-11-02 06:11:17.206 JupyterHub log:174] 200 GET /hub/static/js/home.js?v=20211102061054 (@::ffff:10.162.2. 18) 1.56ms
[I 2021-11-02 06:11:17.238 JupyterHub log:174] 200 GET /hub/api/users (xxx001@::ffff:10.162.2.18) 29.34ms
[D 2021-11-02 06:11:17.243 JupyterHub log:174] 200 GET /hub/static/js/jhapi.js?v=20211102061054 (@::ffff:10.162.2 .18) 1.67ms
[D 2021-11-02 06:11:17.244 JupyterHub log:174] 200 GET /hub/static/js/utils.js?v=20211102061054 (@::ffff:10.162.2 .18) 0.67ms
[D 2021-11-02 06:11:17.246 JupyterHub log:174] 200 GET /hub/static/components/moment/moment.js?v=20211102061054 ( @::ffff:10.162.2.18) 6.70ms
[D 2021-11-02 06:11:20.565 JupyterHub log:174] 304 GET /hub/home (xxx001@::ffff:10.162.2.18) 64.74ms
[I 2021-11-02 06:11:20.697 JupyterHub log:174] 200 GET /hub/api/users (xxx001@::ffff:10.162.2.18) 41.45ms
[I 2021-11-02 06:11:23.437 JupyterHub login:43] User logged out: xxx001
[I 2021-11-02 06:11:23.480 JupyterHub log:174] 302 GET /hub/logout -> /hub/login (xxx001@::ffff:10.162.2.18) 54.6 4ms
[I 2021-11-02 06:11:23.515 JupyterHub log:174] 200 GET /hub/login (@::ffff:10.162.2.18) 6.75ms
[D 2021-11-02 06:11:33.389 JupyterHub ldapauthenticator:379] Attempting to bind xxx001with CN=xxx001,OU=Users,OU =People,OU=SE,OU=***,DC=******,DC=***,DC=biz
[D 2021-11-02 06:11:34.079 JupyterHub ldapauthenticator:392] Status of user bind xxx001with CN=xxx001,OU=Users,O U=People,OU=SE,OU=***,DC=******,DC=***,DC=biz : True
[D 2021-11-02 06:11:34.084 JupyterHub base:482] Setting cookie for xxx001: jupyterhub-services
[D 2021-11-02 06:11:34.084 JupyterHub base:478] Setting cookie jupyterhub-services: {'httponly': True, 'path': '/ services'}
[D 2021-11-02 06:11:34.085 JupyterHub base:478] Setting cookie jupyterhub-session-id: {'httponly': True}
[D 2021-11-02 06:11:34.085 JupyterHub base:482] Setting cookie for xxx001: jupyterhub-hub-login
[D 2021-11-02 06:11:34.085 JupyterHub base:478] Setting cookie jupyterhub-hub-login: {'httponly': True, 'path': ' /hub/'}
[I 2021-11-02 06:11:34.085 JupyterHub base:663] User logged in: xxx001
[I 2021-11-02 06:11:34.086 JupyterHub log:174] 302 POST /hub/login?next= -> /hub/home (xxx001@::ffff:10.162.2.18) 698.01ms
[D 2021-11-02 06:11:34.471 JupyterHub log:174] 304 GET /hub/home (xxx001@::ffff:10.162.2.18) 364.68ms
[I 2021-11-02 06:11:34.587 JupyterHub log:174] 200 GET /hub/api/users (xxx001@::ffff:10.162.2.18) 18.62ms
[D 2021-11-02 06:11:51.696 JupyterHub pages:165] Triggering spawn with default options for xxx001
[D 2021-11-02 06:11:51.696 JupyterHub base:780] Initiating spawn for xxx001
[D 2021-11-02 06:11:51.697 JupyterHub base:787] 0/100 concurrent spawns
[D 2021-11-02 06:11:51.697 JupyterHub base:792] 0 active servers
[D 2021-11-02 06:11:51.718 JupyterHub user:542] Calling Spawner.start for xxx001
[I 2021-11-02 06:11:51.933 JupyterHub mlhubspawner:283] Create network mlhub-xxx001 with subnet 172.33.1.0/24
[E 2021-11-02 06:11:51.949 JupyterHub mlhubspawner:298] Could not connect mlhub to the network and, thus, cannot create the container.
[D 2021-11-02 06:11:52.012 JupyterHub dockerspawner:811] Getting container 'ws-xxx001-mlhub'
[I 2021-11-02 06:11:52.016 JupyterHub dockerspawner:818] Container 'ws-xxx001-mlhub' is gone
[I 2021-11-02 06:11:52.056 JupyterHub mlhubspawner:264] Network mlhub-xxx001 already exists
[E 2021-11-02 06:11:52.060 JupyterHub mlhubspawner:298] Could not connect mlhub to the network and, thus, cannot create the container.
[D 2021-11-02 06:11:52.089 JupyterHub dockerspawner:907] Starting host with config: {'binds': {}, 'links': {}, 'network_mode': 'mlhub-xxx001'}
[I 2021-11-02 06:11:52.161 JupyterHub dockerspawner:1028] Created container ws-xxx001-mlhub (id: 609b2b5) from image localhost/dlab_mlworkspace
[I 2021-11-02 06:11:52.161 JupyterHub dockerspawner:1051] Starting container ws-xxx001-mlhub (id: 609b2b5)
[D 2021-11-02 06:11:52.680 JupyterHub spawner:1084] Polling subprocess every 30s
[I 2021-11-02 06:11:52.726 JupyterHub log:174] 302 GET /hub/spawn/xxx001 -> /hub/spawn-pending/xxx001 (xxx001@::ffff:10.162.2.18) 1035.21ms
[I 2021-11-02 06:11:52.831 JupyterHub pages:303] xxx001 is pending spawn
[I 2021-11-02 06:11:52.836 JupyterHub log:174] 200 GET /hub/spawn-pending/xxx001 (xxx001@::ffff:10.162.2.18) 74.55ms
[W 2021-11-02 06:11:56.881 JupyterHub utils:215] Server at http://172.33.1.2:8080/user/xxx001/ responded with error: 502
[W 2021-11-02 06:11:57.031 JupyterHub utils:215] Server at http://172.33.1.2:8080/user/xxx001/ responded with error: 502
[D 2021-11-02 06:11:57.385 JupyterHub app:1812] Managed service cleanup-service running at http://127.0.0.1:9000
[W 2021-11-02 06:12:01.478 JupyterHub utils:215] Server at http://172.33.1.2:8080/user/xxx001/ responded with error: 502
[D 2021-11-02 06:12:01.701 JupyterHub dockerspawner:811] Getting container 'ws-xxx001-mlhub'
[D 2021-11-02 06:12:01.715 JupyterHub dockerspawner:796] Container 609b2b5 status: {'Dead': False,
'Error': '',
'ExitCode': 0,
'FinishedAt': '0001-01-01T00:00:00Z',
'Health': {'FailingStreak': 0, 'Log': None, 'Status': ''},
'OOMKilled': False,
'Paused': False,
'Pid': 17099,
'Restarting': False,
'Running': True,
'StartedAt': '2021-11-02T06:11:52.563936822Z',
'Status': 'running'}
[W 2021-11-02 06:12:01.715 JupyterHub base:932] User xxx001 is slow to become responsive (timeout=10)
[D 2021-11-02 06:12:01.715 JupyterHub base:937] Expecting server for xxx001 at: http://172.33.1.2:8080/user/xxx001/
[W 2021-11-02 06:12:06.486 JupyterHub utils:215] Server at http://172.33.1.2:8080/user/xxx001/ responded with error: 502
[W 2021-11-02 06:12:11.493 JupyterHub utils:215] Server at http://172.33.1.2:8080/user/xxx001/ responded with error: 502
[W 2021-11-02 06:12:16.498 JupyterHub utils:215] Server at http://172.33.1.2:8080/user/xxx001/ responded with error: 502
[W 2021-11-02 06:12:21.502 JupyterHub utils:215] Server at http://172.33.1.2:8080/user/xxx001/ responded with error: 502
Config
"""
Basic configuration file for jupyterhub.
"""
import os
import re
import signal
import socket
import sys
import docker.errors
import json
from traitlets.log import get_logger
logger = get_logger()
from mlhubspawner import utils
from subprocess import call
c = get_config()
c.Application.log_level = 'DEBUG'
c.Spawner.debug = True
# Override the Jupyterhub `normalize_username` function to remove problematic characters from the username - independent from the used authenticator.
# E.g. when the username is "lastname, firstname" and the comma and whitespace are not removed, they are encoded by the browser, which can lead to brok$
# especially for the tools-part.
# Everybody who starts the hub can override this behavior the same way we do in a mounted `jupyterhub_user_config.py` (Docker local) or via the `hub.ex$
from jupyterhub.auth import Authenticator
original_normalize_username = Authenticator.normalize_username
def custom_normalize_username(self, username):
username = original_normalize_username(self, username)
more_than_one_forbidden_char = False
for forbidden_username_char in [" ", ",", ";", ".", "-", "@", "_"]:
# Replace special characters with a non-special character. Cannot just be empty, like "", because then it could happen that two distinct user n$
# Example: "foo, bar" and "fo, obar" would both become "foobar".
replace_char = "0"
# If there is more than one special character, just replace one of them. Otherwise, "foo, bar" would become "foo00bar" instead of "foo0bar"
if more_than_one_forbidden_char == True:
replace_char = ""
temp_username = username
username = username.replace(forbidden_username_char, replace_char, 1)
if username != temp_username:
more_than_one_forbidden_char = True
return username
Authenticator.normalize_username = custom_normalize_username
original_check_whitelist = Authenticator.check_whitelist
def dynamic_check_whitelist(self, username, authentication=None):
dynamic_whitelist_file = "/resources/users/dynamic_whitelist.txt"
if os.getenv("DYNAMIC_WHITELIST_ENABLED", "false") == "true":
# TODO: create the file and warn the user that the user has to go into the hub pod and modify it there
if not os.path.exists(dynamic_whitelist_file):
logger.error("The dynamic white list has to be mounted to '{}'. Use standard JupyterHub whitelist behavior.".format(dynamic_whitelist_file))
else:
with open(dynamic_whitelist_file, "r") as f:
#whitelisted_users = f.readlines()
# rstrip() will remove trailing whitespaces or newline characters
whitelisted_users = [line.rstrip() for line in f]
return username.lower() in whitelisted_users
return original_check_whitelist(self, username, authentication)
Authenticator.check_whitelist = dynamic_check_whitelist
### Helper Functions ###
def get_or_init(config: object, config_type: type) -> object:
if not isinstance(config, config_type):
return config_type()
return config
def combine_config_dicts(*configs) -> dict:
combined_config = {}
for config in configs:
if not isinstance(config, dict):
config = {}
combined_config.update(config)
return combined_config
### END HELPER FUNCTIONS###
ENV_NAME_HUB_NAME = 'HUB_NAME'
ENV_HUB_NAME = os.environ[ENV_NAME_HUB_NAME]
ENV_EXECUTION_MODE = os.environ[utils.ENV_NAME_EXECUTION_MODE]
# User containers will access hub by container name on the Docker network
c.JupyterHub.hub_ip = '0.0.0.0' #'research-hub
c.JupyterHub.port = 8000
# Persist hub data on volume mounted inside container
# TODO: should really be persisted?
data_dir = os.environ.get('DATA_VOLUME_CONTAINER', '/data')
if not os.path.exists(data_dir):
os.makedirs(data_dir)
c.JupyterHub.cookie_secret_file = os.path.join(data_dir, 'jupyterhub_cookie_secret')
c.JupyterHub.db_url = os.path.join(data_dir, 'jupyterhub.sqlite')
c.JupyterHub.admin_access = True
# prevents directly opening your workspace after login
c.JupyterHub.redirect_to_server=False
c.JupyterHub.allow_named_servers = True
c.Spawner.port = int(os.getenv("DEFAULT_WORKSPACE_PORT", 8080))
# Set default environment variables used by our ml-workspace container
default_env = {"AUTHENTICATE_VIA_JUPYTER": "true", "SHUTDOWN_INACTIVE_KERNELS": "true"}
# Workaround to prevent api problems
#c.Spawner.will_resume = True
# --- Spawner-specific ----
c.JupyterHub.spawner_class = 'mlhubspawner.MLHubDockerSpawner' # override in your config if you want to have a different spawner. If it is the or inher$
#c.Spawner.image = "mltooling/ml-workspace:0.8.7"
#c.Spawner.workspace_images = [c.Spawner.image, "mltooling/ml-workspace-gpu:0.8.7", "mltooling/ml-workspace-r:0.8.7", "mltooling/ml-workspace-spark:0.8$
c.Spawner.image = 'localhost/dlab_mlworkspace'
c.Spawner.workspace_images = [c.Spawner.image]
c.Spawner.notebook_dir = '/workspace'
# Connect containers to this Docker network
c.Spawner.use_internal_ip = True
c.Spawner.prefix = 'ws'
c.Spawner.name_template = c.Spawner.prefix + '-{username}-' + ENV_HUB_NAME + '{servername}' # override in your config when you want to have a different$
# Don't remove containers once they are stopped - persist state
c.Spawner.remove_containers = False
c.Spawner.start_timeout = 600 # should remove errors related to pulling Docker images (see https://github.com/jupyterhub/dockerspawner/issues/293)
c.Spawner.http_timeout = 120
# --- Authenticator ---
c.Authenticator.admin_users = {"xxx001"} # override in your config when needed, for example if you use a different authenticator (e.g. set Github usern$
# Forbid user names that could collide with a named server (check ) to prevent security & routing problems
c.Authenticator.username_pattern = '^((?!-hub).)*$'
NATIVE_AUTHENTICATOR_CLASS = 'nativeauthenticator.NativeAuthenticator'
#c.JupyterHub.authenticator_class = NATIVE_AUTHENTICATOR_CLASS # override in your config if you want to use a different authenticator
c.JupyterHub.authenticator_class = 'ldapauthenticator.LDAPAuthenticator'
c.LDAPAuthenticator.server_address = 'ldap://***.***.**'
c.LDAPAuthenticator.lookup_dn = False
c.LDAPAuthenticator.bind_dn_template = [
'CN={username},OU=Users,OU=People,OU=SE,OU=***,DC=******,DC=***,DC=biz',
'CN={username},OU=Impersonal Accounts,OU=Admin,OU=Central,OU=***,DC=******,DC=***,DC=biz',
]
# --- Load user config ---
# Allow passing an additional config upon mlhub container startup.
# Enables the user to override all configurations occurring above the load_subconfig command; be careful to not break anything ;)
# An empty config file already exists in case the user does not mount another config file.
# The extra config could look like:
# jupyterhub_user_config.py
# > c = get_config()
# > c.DockerSpawner.extra_create_kwargs.update({'labels': {'foo': 'bar'}})
# See https://traitlets.readthedocs.io/en/stable/config.html#configuration-files-inheritance
load_subconfig("{}/jupyterhub_user_config.py".format(os.getenv("_RESOURCES_PATH")))
c.Spawner.environment = get_or_init(c.Spawner.environment, dict)
c.Spawner.environment.update(default_env)
service_environment = {
ENV_NAME_HUB_NAME: ENV_HUB_NAME,
utils.ENV_NAME_EXECUTION_MODE: ENV_EXECUTION_MODE,
utils.ENV_NAME_CLEANUP_INTERVAL_SECONDS: os.getenv(utils.ENV_NAME_CLEANUP_INTERVAL_SECONDS),
}
# shm_size can only be set for Docker, not Kubernetes (see https://stackoverflow.com/questions/43373463/how-to-increase-shm-size-of-a-kubernetes-contai$
#c.Spawner.extra_host_config = { 'shm_size': '256m' }
client_kwargs = {**get_or_init(c.Spawner.client_kwargs, dict)} # {**get_or_init(c.DockerSpawner.client_kwargs, dict), **get_or_init(c.MLHubDockerSpawne$
tls_config = {**get_or_init(c.Spawner.tls_config, dict)} # {**get_or_init(c.DockerSpawner.tls_config, dict), **get_or_init(c.MLHubDockerSpawner.tls_con$
#docker_client = utils.init_docker_client(client_kwargs, tls_config)
#try:
# container = docker_client.containers.list(filters={"id": socket.gethostname()})[0]
# if container.name.lower() != ENV_HUB_NAME.lower():
# container.rename(ENV_HUB_NAME.lower())
#except docker.errors.APIError as e:
# logger.error("Could not correctly start MLHub container. " + str(e))
# os.kill(os.getpid(), signal.SIGTERM)
# For cleanup-service
service_environment.update({"DOCKER_CLIENT_KWARGS": json.dumps(client_kwargs), "DOCKER_TLS_CONFIG": json.dumps(tls_config)})
service_host = "127.0.0.1"
# Consider the case where the user-config contains c.DockerSpawner.environment instead of c.Spawner.environment
# c.DockerSpawner.environment = get_or_init(c.DockerSpawner.environment, dict)
# c.Spawner.environment.update(c.DockerSpawner.environment)
#c.MLHubDockerSpawner.hub_name = ENV_HUB_NAME
# Add nativeauthenticator-specific templates
if c.JupyterHub.authenticator_class == NATIVE_AUTHENTICATOR_CLASS:
import nativeauthenticator
# if template_paths is not set yet in user_config, it is of type traitlets.config.loader.LazyConfigValue; in other words, it was not initialized yet
c.JupyterHub.template_paths = get_or_init(c.JupyterHub.template_paths, list)
# if not isinstance(c.JupyterHub.template_paths, list):
# c.JupyterHub.template_paths = []
c.JupyterHub.template_paths.append("{}/templates/".format(os.path.dirname(nativeauthenticator.__file__)))
# TODO: add env variable to readme
if (os.getenv("IS_CLEANUP_SERVICE_ENABLED", "true") == "true"):
c.JupyterHub.services = [
{
'name': 'cleanup-service',
'admin': True,
'url': 'http://{}:9000'.format(service_host),
'environment': service_environment,
'command': [sys.executable, '/resources/cleanup-service.py']
}
]
Hub version mltooling/ml-workspace-minimal:0.13.2, mltooling/ml-hub:1.0.0
I'm trying to run ml-hub with ml-workspace but having problems starting the spawner. The hub is up and running, logging in users (ldap) is working fine but it fails when i start the spawner.
The setup is pretty much standard except that
Log output
Config
The text was updated successfully, but these errors were encountered: