HTTP 500 Error when creating dashboard using traefik

[tstreibl]

Describe the bug
I try to run a dockerized Jupyterhub with ContaindDS Dashboard using traefik as reverse proxy. Everything works fine except the creation of a containds dashbord, failing with a HTTP 500 error. I'm not very experianced so any hint what I'm doing wrong would be welcome.

To Reproduce
I use a .ipynb notebook which I could watch with voila within my user notebook for testing. When I try to use the dashboard interface to turn this testnotebook into a ContainDS Dashboard I get the HTTP 500 error.

Screenshots

Configuration
I used standard Docker Files for Jupyterhub and Jupyterlab (for Jupyterhub: jupyterhub/jupyterhub:2.0.1 or jupyterhub/jupyterhub:1.5.0; for Jupyterlab jupyter/scipy-notebook:hub-2.0.1 or a jupyter/scipy-notebook:hub-1.5.0), added the necessary ContainDS packages ( voila,streamlit,dash,cdsdashboards,jhsingle-native-proxy, jupyter-containds ) via pip install and created images for Jupyterhub and Jupyterlab. Those images are used in a docker_compose container for the Jupyterhub/Jupyterlab services . The traefik reverse proxy is running in a seperate container and provides the network for the jupyter container (Full setup see https://github.com/tstreibl/Jupyterhub_Traefik_ContainDS_Test; readme in doc folder contains more details).

classes used:

• authentication: oauthenticator.LocalGitHubOAuthenticator
• spawner: cdsdashboards.hubextension.spawners.variabledocker.VariableDockerSpawner
• CDSDashboardsConfig.builder_class: cdsdashboards.builder.dockerbuilder.DockerBuilder

During the final build phase (log says "In do_final_build") the containds container fails, because the "jhsingle_native_proxy.main" module could not be found, although I pip installed it in the jupyterhub and in the jupyterlab images. For that reason the containds container never shows up on the network, which results in a timeout and probably the http 500 error.

I have no idea if I used the wrong docker files for jupyterhub/jupyterlab, if I messed up building the images, especially the pip installations or if it's a network issue concerning traefik and the way the traefik network is provided to jupyterhub.

Log from containds dashboard container which gets spawned and fails

Entered start.sh with args: python3 -m jhsingle_native_proxy.main --destport=0 python3 {-}m voila {presentation_path} {--}port={port} {--}no-browser {--}Voila.base_url={base_url}/ {--}Voila.server_url=/ --progressive --presentation-path=/home/jovyan/voila_test.ipynb --ip=0.0.0.0 --port=8888 {--}debug --debug
Granting jovyan passwordless sudo rights!
Running as jovyan: python3 -m jhsingle_native_proxy.main --destport=0 python3 {-}m voila {presentation_path} {--}port={port} {--}no-browser {--}Voila.base_url={base_url}/ {--}Voila.server_url=/ --progressive --presentation-path=/home/jovyan/voila_test.ipynb --ip=0.0.0.0 --port=8888 {--}debug --debug
/opt/conda/bin/python3: Error while finding module specification for 'jhsingle_native_proxy.main' (ModuleNotFoundError: No module named 'jhsingle_native_proxy')

log from jupyterlab container waiting for the containds dashboard container to show up

[I 2021-12-27 21:23:41.370 JupyterHub dockerspawner:1272] Created container jupyter-tstreibl-dash-2dtest-2ddashboard (id: e773946) from image jupyterlab_img
[I 2021-12-27 21:23:41.370 JupyterHub dockerspawner:1296] Starting container jupyter-tstreibl-dash-2dtest-2ddashboard (id: e773946)
[I 2021-12-27 21:23:41.437 JupyterHub log:189] 200 GET /hub/dashboards/test-dashboard (tstreibl@::ffff:46.128.245.7) 25.37ms
[D 2021-12-27 21:23:41.661 JupyterHub log:189] 304 GET /hub/dashboards-static/css/style.css (@::ffff:46.128.245.7) 1.30ms
[D 2021-12-27 21:23:41.810 JupyterHub log:189] 200 GET /hub/dashboards-static/js/viewdashboard.js?v=20211227212041 (@::ffff:46.128.245.7) 2.53ms
[D 2021-12-27 21:23:42.011 JupyterHub spawner:1221] Polling subprocess every 30s
[D 2021-12-27 21:23:51.266 JupyterHub dockerspawner:982] Getting container 'jupyter-tstreibl-dash-2dtest-2ddashboard'
[D 2021-12-27 21:23:51.272 JupyterHub dockerspawner:967] Container e773946 status: {'Dead': False,
     'Error': '',
     'ExitCode': 1,
     'FinishedAt': '2021-12-27T20:23:42.0838958Z',
     'OOMKilled': False,
     'Paused': False,
     'Pid': 0,
     'Restarting': False,
     'Running': False,
     'StartedAt': '2021-12-27T20:23:41.995181933Z',
     'Status': 'exited'}
[D 2021-12-27 21:23:51.273 JupyterHub base:226] In do_final_build
[W 2021-12-27 21:23:51.273 JupyterHub web:1787] 500 GET /hub/dashboards-api/test-dashboard/progress (::ffff:46.128.245.7): Spawner failed to start [status=ExitCode=1, Error='', FinishedAt=2021-12-27T20:23:42.0838958Z]. The logs for tstreibl:dash-test-dashboard may contain details.
[E 2021-12-27 21:23:51.274 JupyterHub web:1197] Cannot send error response after headers written
[I 2021-12-27 21:23:51.275 JupyterHub log:189] 200 GET /hub/dashboards-api/test-dashboard/progress (tstreibl@::ffff:46.128.245.7) 9393.61ms
[I 2021-12-27 21:23:56.422 JupyterHub log:189] 200 GET /hub/dashboards-api/test-dashboard/progress (tstreibl@::ffff:46.128.245.7) 10.12ms
[D 2021-12-27 21:24:07.639 JupyterHub dockerspawner:982] Getting container 'jupyter-tstreibl-'
[D 2021-12-27 21:24:07.661 JupyterHub dockerspawner:967] Container a840df8 status: {'Dead': False,
     'Error': '',
     'ExitCode': 0,
     'FinishedAt': '0001-01-01T00:00:00Z',
     'OOMKilled': False,
     'Paused': False,
     'Pid': 197417,
     'Restarting': False,
     'Running': True,
     'StartedAt': '2021-12-27T20:21:37.61964646Z',
     'Status': 'running'}
[D 2021-12-27 21:24:12.015 JupyterHub dockerspawner:982] Getting container 'jupyter-tstreibl-dash-2dtest-2ddashboard'
[D 2021-12-27 21:24:12.021 JupyterHub dockerspawner:967] Container e773946 status: {'Dead': False,
     'Error': '',
     'ExitCode': 1,
     'FinishedAt': '2021-12-27T20:23:42.0838958Z',
     'OOMKilled': False,
     'Paused': False,
     'Pid': 0,
     'Restarting': False,
     'Running': False,
     'StartedAt': '2021-12-27T20:23:41.995181933Z',
     'Status': 'exited'}
[W 2021-12-27 21:24:22.376 JupyterHub user:811] tstreibl's server never showed up at http://192.168.144.4:8888/user/tstreibl/dash-test-dashboard/ after 30 seconds. Giving up. 

Common causes of this timeout, and debugging tips:
    
    1. The server didn't finish starting,
       or it crashed due to a configuration issue.
       Check the single-user server's logs for hints at what needs fixing.
    2. The server started, but is not accessible at the specified URL.
       This may be a configuration issue specific to your chosen Spawner.
       Check the single-user server logs and resource to make sure the URL
       is correct and accessible from the Hub.
    3. (unlikely) Everything is working, but the server took too long to respond.
       To fix: increase `Spawner.http_timeout` configuration
       to a number of seconds that is enough for servers to become responsive.

[D 2021-12-27 21:24:22.377 JupyterHub user:860] Stopping tstreibl:dash-test-dashboard
[D 2021-12-27 21:24:22.377 JupyterHub dockerspawner:982] Getting container 'jupyter-tstreibl-dash-2dtest-2ddashboard'
[D 2021-12-27 21:24:22.384 JupyterHub dockerspawner:967] Container e773946 status: {'Dead': False,
     'Error': '',
     'ExitCode': 1,
     'FinishedAt': '2021-12-27T20:23:42.0838958Z',
     'OOMKilled': False,
     'Paused': False,
     'Pid': 0,
     'Restarting': False,
     'Running': False,
     'StartedAt': '2021-12-27T20:23:41.995181933Z',
     'Status': 'exited'}
[D 2021-12-27 21:24:22.401 JupyterHub user:883] Finished stopping tstreibl:dash-test-dashboard
[E 2021-12-27 21:24:22.409 JupyterHub gen:623] Exception in Future <Task finished name='Task-136' coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() done, defined at /usr/local/lib/python3.8/dist-packages/jupyterhub/handlers/base.py:935> exception=TimeoutError("Server at http://192.168.144.4:8888/user/tstreibl/dash-test-dashboard/ didn't respond in 30 seconds")> after timeout
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/dist-packages/tornado/gen.py", line 618, in error_callback
        future.result()
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/handlers/base.py", line 942, in finish_user_spawn
        await spawn_future
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/user.py", line 792, in spawn
        await self._wait_up(spawner)
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/user.py", line 836, in _wait_up
        raise e
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/user.py", line 806, in _wait_up
        resp = await server.wait_up(
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/utils.py", line 241, in wait_for_http_server
        re = await exponential_backoff(
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/utils.py", line 189, in exponential_backoff
        raise asyncio.TimeoutError(fail_message)
    asyncio.exceptions.TimeoutError: Server at http://192.168.144.4:8888/user/tstreibl/dash-test-dashboard/ didn't respond in 30 seconds
    
[D 2021-12-27 21:24:37.639 JupyterHub dockerspawner:982] Getting container 'jupyter-tstreibl-'
[D 2021-12-27 21:24:37.645 JupyterHub dockerspawner:967] Container a840df8 status: {'Dead': False,
     'Error': '',
     'ExitCode': 0,
     'FinishedAt': '0001-01-01T00:00:00Z',
     'OOMKilled': False,
     'Paused': False,
     'Pid': 197417,
     'Restarting': False,
     'Running': True,
     'StartedAt': '2021-12-27T20:21:37.61964646Z',
     'Status': 'running'}
[D 2021-12-27 21:25:07.638 JupyterHub dockerspawner:982] Getting container 'jupyter-tstreibl-'
[D 2021-12-27 21:25:37.645 JupyterHub dockerspawner:967] Container a840df8 status: {'Dead': False,
     'Error': '',
     'ExitCode': 0,
     'FinishedAt': '0001-01-01T00:00:00Z',
     'OOMKilled': False,
     'Paused': False,
     'Pid': 197417,
     'Restarting': False,
     'Running': True,
     'StartedAt': '2021-12-27T20:21:37.61964646Z',
     'Status': 'running'}
[D 2021-12-27 21:25:42.094 JupyterHub proxy:821] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
21:25:42.100 [ConfigProxy] info: 200 GET /api/routes 
[D 2021-12-27 21:25:42.111 JupyterHub proxy:346] Checking routes

modules installed in the jupyterlab container

(base) root@d815dc72447d:~# python3 -m pip list
WARNING: The directory '/home/jovyan/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag.
Package                       Version
----------------------------- -----------
aiohttp                       3.8.1
aiosignal                     1.2.0
alembic                       1.7.5
altair                        4.1.0
anyio                         3.3.4
appdirs                       1.4.4
argon2-cffi                   21.1.0
astor                         0.8.1
async-generator               1.10
async-timeout                 4.0.2
attrs                         21.2.0
Babel                         2.9.1
backcall                      0.2.0
backports.functools-lru-cache 1.6.4
base58                        2.1.1
beautifulsoup4                4.10.0
bleach                        4.1.0
blinker                       1.4
bokeh                         2.4.1
Bottleneck                    1.3.2
bqplot                        0.12.31
Brotli                        1.0.9
brotlipy                      0.7.0
cached-property               1.5.2
cachetools                    5.0.0
cdsdashboards                 0.6.0
certifi                       2021.10.8
certipy                       0.1.3
cffi                          1.15.0
chardet                       4.0.0
charset-normalizer            2.0.0
click                         7.1.2
cloudpickle                   2.0.0
colorama                      0.4.4
conda                         4.10.3
conda-package-handling        1.7.3
cryptography                  35.0.0
cycler                        0.11.0
Cython                        0.29.24
cytoolz                       0.11.2
dash                          2.0.0
dash-core-components          2.0.0
dash-html-components          2.0.0
dash-table                    5.0.0
dask                          2021.11.2
debugpy                       1.5.1
decorator                     5.1.0
defusedxml                    0.7.1
dill                          0.3.4
distributed                   2021.11.2
entrypoints                   0.3
Flask                         2.0.2
Flask-Compress                1.10.1
fonttools                     4.28.1
frozenlist                    1.2.0
fsspec                        2021.11.0
gitdb                         4.0.9
GitPython                     3.1.24
gmpy2                         2.1.0rc1
greenlet                      1.1.2
h5py                          3.4.0
HeapDict                      1.0.1
idna                          3.1
imagecodecs                   2021.8.26
imageio                       2.9.0
importlib-metadata            4.8.2
importlib-resources           5.4.0
ipykernel                     6.5.0
ipympl                        0.8.2
ipython                       7.29.0
ipython-genutils              0.2.0
ipywidgets                    7.6.5
itsdangerous                  2.0.1
jedi                          0.18.1
jhsingle-native-proxy         0.8.0
Jinja2                        3.0.3
joblib                        1.1.0
json5                         0.9.5
jsonschema                    4.2.1
jupyter-client                7.0.6
jupyter-containds             0.2.2
jupyter-core                  4.9.1
jupyter-server                1.11.2
jupyter-telemetry             0.1.0
jupyterhub                    1.5.0
jupyterlab                    3.2.4
jupyterlab-pygments           0.1.2
jupyterlab-server             2.8.2
jupyterlab-widgets            1.0.2
kiwisolver                    1.3.2
libmambapy                    0.18.1
llvmlite                      0.37.0
locket                        0.2.0
Mako                          1.1.6
mamba                         0.18.1
MarkupSafe                    2.0.1
matplotlib                    3.5.0
matplotlib-inline             0.1.3
mistune                       0.8.4
mock                          4.0.3
mpmath                        1.2.1
msgpack                       1.0.2
multidict                     5.2.0
munkres                       1.1.4
nbclassic                     0.3.4
nbclient                      0.5.9
nbconvert                     6.3.0
nbformat                      5.1.3
nest-asyncio                  1.5.1
networkx                      2.6.3
notebook                      6.4.6
numba                         0.54.1
numexpr                       2.7.3
numpy                         1.20.3
oauthlib                      3.1.1
olefile                       0.46
packaging                     21.3
pamela                        1.0.0
pandas                        1.3.4
pandocfilters                 1.5.0
parso                         0.8.2
partd                         1.2.0
patsy                         0.5.2
pexpect                       4.8.0
pickleshare                   0.7.5
Pillow                        8.4.0
pip                           21.3.1
plotly                        5.5.0
pluggy                        1.0.0
pooch                         1.5.2
prometheus-client             0.12.0
prompt-toolkit                3.0.22
protobuf                      3.19.1
psutil                        5.8.0
ptyprocess                    0.7.0
pyarrow                       6.0.1
pycosat                       0.6.3
pycparser                     2.21
pycurl                        7.44.1
pydeck                        0.7.1
Pygments                      2.10.0
PyJWT                         2.3.0
Pympler                       1.0.1
pyOpenSSL                     21.0.0
pyparsing                     3.0.6
pyrsistent                    0.18.0
PySocks                       1.7.1
python-dateutil               2.8.2
python-json-logger            2.0.1
pytz                          2021.3
pytz-deprecation-shim         0.1.0.post0
PyWavelets                    1.2.0
PyYAML                        6.0
pyzmq                         22.3.0
requests                      2.26.0
ruamel.yaml                   0.17.17
ruamel.yaml.clib              0.2.6
ruamel-yaml-conda             0.15.80
scikit-image                  0.18.3
scikit-learn                  1.0.1
scipy                         1.7.2
seaborn                       0.11.2
Send2Trash                    1.8.0
setuptools                    59.2.0
simpervisor                   0.4
six                           1.16.0
smmap                         5.0.0
sniffio                       1.2.0
sortedcontainers              2.4.0
soupsieve                     2.3
SQLAlchemy                    1.4.27
statsmodels                   0.13.1
streamlit                     1.3.1
sympy                         1.9
tables                        3.6.1
tblib                         1.7.0
tenacity                      8.0.1
terminado                     0.12.1
testpath                      0.5.0
threadpoolctl                 3.0.0
tifffile                      2021.11.2
toml                          0.10.2
toolz                         0.11.2
tornado                       6.1
tqdm                          4.62.3
traitlets                     5.1.1
traittypes                    0.2.1
typing_extensions             4.0.0
tzdata                        2021.5
tzlocal                       4.1
urllib3                       1.26.7
validators                    0.18.2
voila                         0.3.0
watchdog                      2.1.6
wcwidth                       0.2.5
webencodings                  0.5.1
websocket-client              1.2.1
websockets                    10.1
Werkzeug                      2.0.2
wheel                         0.37.0
widgetsnbextension            3.5.2
xlrd                          2.0.1
yarl                          1.7.2
zict                          2.0.0
zipp                          3.6.0

Docker File to create the jupyterhub image

# see https://hub.docker.com/r/jupyterhub/jupyterhub/tags
#FROM jupyterhub/jupyterhub:1.5.0
FROM jupyterhub/jupyterhub:2.0.1

COPY jupyterhub_config.py .

#Example see https://stackoverflow.com/questions/27701930/how-to-add-users-to-docker-container 
#system user tstreibl and normal user testuser
RUN useradd -r -m -s /bin/bash -g users -G sudo,users -p meinpasswort tstreibl 
#RUN useradd -m -s /bin/bash -g users -p meinpasswort tstreibl 
RUN useradd -m -s /bin/bash -g users -p testuserpasswort testuser                           

USER root
RUN apt-get -y update && apt-get install 

# voila-gridstack would be an option
RUN pip3 install --no-cache-dir \
    dockerspawner \
    voila \
    streamlit \
    dash \
	cdsdashboards \
	jhsingle-native-proxy \
    jupyter-containds \
    oauthenticator 

#USER tstreibl  # does not work, seems to be necessary that at the end user is root 

Docker File to create the jupyterlab image

#see https://hub.docker.com/r/jupyter/scipy-notebook/tags
#FROM jupyter/scipy-notebook:hub-1.5.0
FROM jupyter/scipy-notebook:hub-2.0.1
#FROM jupyter/scipy-notebook:latest

USER root

# vermutlich braucht man keines der Zusatzpackages; sagemath ist aber ein nettes addon
# APT packages   
RUN apt-get update && \
    apt-get install -y --no-install-recommends \
    fonts-dejavu \
    tzdata \
    gfortran \
    gcc \
    scilab \
    pari-gp \
    libpari-dev \
    sagemath \
    sagemath-jupyter \
    libgmp-dev \
    && apt-get clean && \
    rm -rf /var/lib/apt/lists/*


USER $NB_UID
#tornado is already installed
#jhsingle-native-proxy seems to be required for cdsdashboards but is not automatically installed 
RUN pip3 install --no-cache-dir \
    voila \
    streamlit \
    dash \
    cdsdashboards \
    jupyter-containds \
    jhsingle-native-proxy \
    bqplot  && \
    fix-permissions "${CONDA_DIR}" && \
    fix-permissions "/home/${NB_USER}" 

#  RUN conda install --quiet --yes \
#     tornado \
#     voila \
#     streamlit \
#     dash \
#     cdsdashboards \
#     jupyter-containds \
#     jhsingle-native-proxy \
#     bqplot  && \
#     fix-permissions $CONDA_DIR

USER $NB_UID  #seems to be mandatory that the user is not root at the end

Docker-Compose file for the jpyterhub and jupyterlab services

version: "3.7"

networks:
  #Enable connection with Traefik 
  traefik:
    external: true
  jupyter_net:  # entspricht ${INTERNAL_NETWORK}
    name: "jupyter_net"
    external: false
    driver: bridge  # ohne den Bridge Mode funktioniert gar nichts

volumes:
  jupyterhub_data:
  jupyterhub_config:
services:

  jupyterhub:

    container_name: jupyterhub
    
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - jupyterhub_data:/srv/jupyterhub # this volume must be deleted when changing jupyterhub_config.py otherwise changes won't reflect
      - jupyterhub_config:/etc/jupyterhub # all configurations
      - "/home/tilo/traefik/letsencrypt/certs/jupyter.tilostreibl.de:/certs:ro"
    
   
    networks:   
      - traefik
      - ${INTERNAL_NETWORK} # - jupyter_net
     
    ports:
      - "8000:8000"
     # - "8001:8001"
     # - "8081:8081"
      - "8888:8888" # necessary?
     
    env_file: 
      - ./.env
    environment:
      TZ: Europe/Stockholm #set the time zone in the container     
     
      HOST: "${HOST}"   
      COMPOSE_PROJECT_NAME: "${COMPOSE_PROJECT_NAME}"
      DOCKER_JUPYTER_CONTAINER: "${DOCKER_JUPYTER_CONTAINER}" #must match the image name build in jupyterlab service
      DOCKER_NOTEBOOK_DIR: "${DOCKER_NOTEBOOK_DIR}" # directory im Container innerhalb des für jupyterhub generierten binds, wo die jupyter notebooks der User gespeichert werden      
      #- DOCKER_JUPYTER_CONTAINER=jupyterlab_img
      HUB_IP: jupyterhub   #must match container_name  (0.0.0.0 does not work)

    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=${INTERNAL_NETWORK}"
      - "traefik.frontend.rule=Host:${HOST}" 
      - "providers.docker=true"
      
      # Route HTTPS
      - "traefik.http.routers.jupyterhub-secure.entrypoints=web-secure"
      - "traefik.http.routers.jupyterhub-secure.rule=Host(`${HOST}`)"
      
      # Enable TLS
      - "traefik.http.routers.jupyterhub-secure.tls=true"
      # Enable Let's Encrypt HTTP challenge
      - "traefik.http.routers.traefik-secure-secured.tls.certresolver=letsencrypthttpchallenge"
      - "traefik.http.routers.jupyterhub-secure.tls.certresolver=letsencrypthttpchallenge"      
  
    image: jupyterhub_img:latest
    build: 
      context: ./jupyterhub
    restart: on-failure

  jupyterlab:
    container_name: jupyterlab-throaway
    build: 
      context: ./jupyterlab    
    restart: on-failure
    image: ${DOCKER_JUPYTER_CONTAINER}:latest
    networks:   
      - ${INTERNAL_NETWORK}
    command: echo  # to stop the container being run from docker - the hub will start the container
    env_file: 
      - ./.env
    environment:
      TZ: Europe/Stockholm #set the time zone in the container
      DOCKER_NOTEBOOK_DIR: "${DOCKER_NOTEBOOK_DIR}" # directory im Container innerhalb des für jupyterhub generierten binds, wo die jupyter notebooks der User gespeichert werden 
      GRANT_SUDO: 'yes'
      CHOWN_HOME: 'yes'
      CHOWN_HOME_OPTS: '-R'
      JUPYTERHUB_ANYONE: 1
      JUPYTER_ENABLE_LAB: 'yes' #see https://github.com/jupyter/docker-stacks/issues/1217
      NB_GID: 1000 # entspricht Group "users", der default eingerichtete javyan user gehört zu dieser Gruppe
    labels:     
      # Connect to Traefik
      - "traefik.enable=true"
      - "traefik.frontend.rule=Host:${HOST}" 
      - "traefik.docker.network= ${INTERNAL_NETWORK}"
      # Route HTTPS
      - "traefik.http.routers.jupyterlab-secure.entrypoints=web-secure"
      - "traefik.http.routers.jupyterlab-secure.rule=Host(`${HOST}`)"
      # Enable TLS
      - "traefik.http.routers.jupyterlab-secure.tls=true"
      # Enable Let's Encrypt HTTP challenge
      - "traefik.http.routers.jupyterlab-secure.tls.certresolver=letsencrypthttpchallenge"
      # Enable HSTS headers
      - "traefik.http.middlewares.hstsheaders.headers.stsSeconds=315360000"
      - "traefik.http.middlewares.hstsheaders.headers.forceSTSHeader=true"
      - "traefik.http.middlewares.hstsheaders.headers.stsPreload=true"
      - "traefik.http.middlewares.hstsheaders.headers.stsIncludeSubdomains=true"

Docker-Compose file for the traefic reverse proxy

version: "3.7"

networks:
    #Allow back-ends to communicate with this
    traefik:
        external: true
        name: traefik 
        
services:
  traefik:
    image: "traefik:latest"
    env_file: 
      - ./.env
    container_name: "traefik"
    restart: always
    command:
      #Only if you want to use the Traefik Dashboard
      - "--log.level=DEBUG"
      - "--api.insecure=false"  
      - "--api.dashboard=true"
      # Define Docker as the provider here
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      #The location of where we will store our dynamic file with various configurations possible "on the fly"
      - "--providers.file.directory=/etc/traefik/dynamic"
      #Entrypoint -> is a front-end named "web" on HTTP Port 80
      - "--entrypoints.web.address=:80"
      #Entrypoint -> is a front-end named "web-secure" on HTTPS Port 443
      - "--entrypoints.web-secure.address=:443"
      #HTTP challenge as our method to acquire SSL certificates
      - "--certificatesresolvers.letsencrypthttpchallenge.acme.httpchallenge=true"
      - "--certificatesresolvers.letsencrypthttpchallenge.acme.httpchallenge.entrypoint=web"
      #Define Let's Encrypt as the issuer
      #- "--certificatesresolvers.letsencrypthttpchallenge.acme.email=webmaster@tilostreibl.de"
      - "--certificatesresolvers.letsencrypthttpchallenge.acme.email=${EMAIL}"
      #- "--certificatesresolvers.letsencrypthttpchallenge.acme.onhostrule=true"
      # Location of the acme.json file
      - "--certificatesresolvers.letsencrypthttpchallenge.acme.storage=/letsencrypt/acme.json"
    labels:
      - "traefik.enable=true"
      - "traefik.frontend.rule=Host:${DOMAIN}"
      - "traefik.docker.network=traefik"     
       #This will enable SSL
      - "traefik.http.routers.traefik-secure.tls=true"
      #Enable Let's Encrypt HTTP challenge
      - "traefik.http.routers.traefik-secure.tls.certresolver=letsencrypthttpchallenge"
      #Use the web-secure as our entrypoint at 443
      - "traefik.http.routers.traefik-secure.entrypoints=web-secure"
      #The router here is just called traefik-secure and the FQDN rule
      - "traefik.http.routers.traefik-secure.rule=Host(`${DASHBOARD}`)"
      #Enable traefik-auth middleware to use
      - "traefik.http.routers.traefik-secure.middlewares=traefik-auth"
      #Refer to traefik's dashboard service
      - "traefik.http.routers.traefik-secure.service=api@internal"
      #Hashed Password for the dashboard
      - "traefik.http.middlewares.traefik-auth.basicauth.users=admin:${ADMIN_PASS}"
      #Global redirection: http to https
      - traefik.http.routers.http-catchall.rule=HostRegexp(`{host:(www\.)?.+}`)
      - traefik.http.routers.http-catchall.entrypoints=web
      - traefik.http.routers.http-catchall.middlewares=wwwtohttps
      #Global redirection: https (www.) to https
      - traefik.http.routers.wwwsecure-catchall.rule=HostRegexp(`{host:(www\.).+}`)
      - traefik.http.routers.wwwsecure-catchall.entrypoints=web-secure
      - traefik.http.routers.wwwsecure-catchall.tls=true
      - traefik.http.routers.wwwsecure-catchall.middlewares=wwwtohttps
      #middleware: http(s)://(www.) to  https://
      - traefik.http.middlewares.wwwtohttps.redirectregex.regex=^https?://(?:www\.)?(.+)
      - traefik.http.middlewares.wwwtohttps.redirectregex.replacement=https://$${1}
      - traefik.http.middlewares.wwwtohttps.redirectregex.permanent=true
      
    #Specifying the Ports Traefik will listen on
    ports:
      - "80:80"
      - "443:443"
    #The network Traefik will use is just traefik
    networks:
      - "traefik"
    volumes:
      #Docker socket
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      #Let's Encrypt certificates in ./letsencrypt/acme.json
      - "./letsencrypt:/letsencrypt"
      #Mount the dynamic file configuration
      - ./dynamic-config.yaml:/etc/traefik/dynamic/dynamic-config.yaml

  #wandelt das Letsencrypt Zertifikat von Traefic aus der Datei acme.json in ander Zertifikat Formate im Verzeichnis dump um
  #wird für jupyter gebraucht, weil man dort keine acme.json einlesen kann, sondern die einzelnen privatekey.key und certificate.crt benötigt
  #Details: https://github.com/DanielHuisman/traefik-certificate-extractor
  #Details: https://github.com/ldez/traefik-certs-dumper/issues/46
  certdumper: 
    image: ldez/traefik-certs-dumper:latest
    logging:  #man kann den Log Output nicht konfiguruieren, daher vorerst diese Methode um die Logs der anderen Apps besser zu sehen
      driver: none
    command: "file --watch --domain-subdir=true --version v2"
    volumes:
      - "./letsencrypt/acme.json:/acme.json:ro"
      - "./letsencrypt/certs:/dump"

jupyterhub_config.py

#JupyterHub configuration
#
##If you update this file, do not forget to delete the `jupyterhub_data` volume before restarting the jupyterhub service:
##
##docker volume rm jupyterhub_jupyterhub_data
##
## or, if you changed the COMPOSE_PROJECT_NAME to <name>:
##
##docker volume rm <name>_jupyterhub_data
##

import os, sys
import shutil

#print shows up in logs with "flush=True"
print("This is a testoutput from jupyterhub_config.py using print",flush=True)

c.JupyterHub.log_level = 10

#Files are created from the certdumper docker service (see docker-compose for traefic)
#traefic and letsencrypt provides automatically an acme.json file
#the certdumper docker service only converts the acme.json file into the key and cert files
#would be much easier if we could load the acme.json directly at this place  
c.JupyterHub.ssl_key = '/certs/privatekey.key'  #aus dem Laufwerk des Containers
c.JupyterHub.ssl_cert = '/certs/certificate.crt'

c.JupyterHub.trusted_downstream_ips = [os.environ['HOST_IP']]  
c.JupyterHub.cleanup_proxy = True
c.JupyterHub.cleanup_servers = True

c.JupyterHub.hub_ip = '0.0.0.0' #os.environ['HUB_IP'] works also
c.JupyterHub.hub_connect_ip = os.environ['HUB_IP']  #mandatory
#Default: 8081 
#c.JupyterHub.hub_port = 8081
 
c.CDSDashboardsConfig.builder_class = 'cdsdashboards.builder.dockerbuilder.DockerBuilder'

##Generic
c.JupyterHub.admin_access = True
c.Spawner.default_url = '/lab'  #{username} will be expanded to the user’s username
c.JupyterHub.shutdown_on_logout = True # von mir; Session nach logout zu machen


##Authenticator

from oauthenticator.github import LocalGitHubOAuthenticator
c.JupyterHub.authenticator_class = 'oauthenticator.LocalGitHubOAuthenticator' # GitHubOAuthenticator

c.Authenticator.auto_login_oauth2_authorize = False
c.Authenticator.oauth_callback_url = os.environ['oauth_callback_url']
c.Authenticator.client_id = os.environ['client_id']
c.Authenticator.client_secret = os.environ['client_secret'] 

c.Authenticator.allowed_users = {'tstreibl','testuser'}
c.Authenticator.admin_users = {'tstreibl'}  
c.Authenticator.create_system_users = True

#Enabling ContainDS Dashboard; see https://cdsdashboards.readthedocs.io/en/stable/chapters/setup/docker.html#installing-cdsdashboards
from cdsdashboards.app import CDS_TEMPLATE_PATHS
from cdsdashboards.hubextension import cds_extra_handlers
#from cdsdashboards.app import CDSDashboardsConfig 

c.JupyterHub.template_paths = CDS_TEMPLATE_PATHS
c.JupyterHub.extra_handlers = cds_extra_handlers

#c.VariableMixin.proxy_force_alive = True
#c.VariableMixin.proxy_last_activity_interval = 300
#c.VariableMixin.proxy_request_timeout = 0

import secrets
os.environ['JPY_COOKIE_SECRET'] = secrets.token_hex(16)
c.JupyterHub.cookie_secret = bytes.fromhex(os.environ['JPY_COOKIE_SECRET'])

c.CDSDashboardsConfig.builder_class = 'cdsdashboards.builder.dockerbuilder.DockerBuilder'
c.CDSDashboardsConfig.allow_custom_conda_env = False
c.CDSDashboardsConfig.include_auth_state = True
c.CDSDashboardsConfig.include_servers = True
c.CDSDashboardsConfig.include_servers_state = True


c.JupyterHub.spawner_class = 'cdsdashboards.hubextension.spawners.variabledocker.VariableDockerSpawner'
c.Spawner.debug = True
c.JupyterHub.allow_named_servers = True

c.Spawner.image = os.environ['DOCKER_JUPYTER_CONTAINER']
#servername = os.environ['HOST'] 
c.Spawner.extra_host_config = {'network_mode': os.environ['INTERNAL_NETWORK']}
c.Spawner.hub_ip_connect =  os.environ['HUB_IP']  
c.Spawner.network_name = os.environ['INTERNAL_NETWORK'] #"jupyter_net"
c.Spawner.use_internal_ip = True
c.Spawner.remove = False # remove user containers when done. True works better but difficult to debug
c.Spawner.debug = True
notebook_dir = os.environ.get('DOCKER_NOTEBOOK_DIR')
c.Spawner.notebook_dir = notebook_dir
c.Spawner.extra_create_kwargs = {'user': 'root'} #jovyan

c.Spawner.environment = {
  'GRANT_SUDO': '1',
  'UID': '0', # workaround https://github.com/jupyter/docker-stacks/pull/420
}

c.Spawner.name_template = "{prefix}-{username}-{servername}"
c.Spawner.volumes = {"{prefix}-{username}-{servername}": notebook_dir }

c.Spawner.cpu_limit = 1
c.Spawner.mem_limit = '10G'

[maarten-betman]

I'm having the same issue with a TLJH deployment on a VM running Ubuntu 20.02.
I've used cdsdashboards==0.5.7 since there is remark in the changelog when user DockerSpawner, but nu luck unfortunately.

[maarten-betman]

I was struggling with the folder rights to write to folder after mounting volumes. I saw a tip somewhere on github (sorry can't find where at the moment) to include a pre_spawn_hook to create folder and alter the ownership. Further it advised to add c.DockerSpawner.extra_create_kwargs = {"user": "root"}. After commenting this out the containers successfully started for dashboards.

[yannsartori]

Something like @maarten-betman's solution kind of worked (in my case, I had spawner.extra_container_spec = {'user': '0'} in my pre_spawn_hook) in that I didn't encounter the "module not found issue". However, this solution was not satisfactory because the files in the server's containers weren't being mounted properly.

It seems that by using root as the user, the conda environment wasn't being sourced, and so the command python3 being used didn't have the jhsingle module. However, the command python does have the jhsingle module. Hence, I added the following command to my jupyterhub_config.py to use python instead of python3:

c.VariableMixin.default_presentation_cmd = ['start.sh', 'python', '-m', 'jhsingle_native_proxy.main']

and it seemed to have resolve it in my case.