Custom envs #370
Custom envs #370
Conversation
frontend fixes to be done
SwanCustomEnvironments/swancustomenvironments/serverextension.py
Outdated
Show resolved
Hide resolved
SwanCustomEnvironments/swancustomenvironments/serverextension.py
Outdated
Show resolved
Hide resolved
SwanCustomEnvironments/swancustomenvironments/serverextension.py
Outdated
Show resolved
Hide resolved
SwanCustomEnvironments/swancustomenvironments/serverextension.py
Outdated
Show resolved
Hide resolved
SwanCustomEnvironments/swancustomenvironments/serverextension.py
Outdated
Show resolved
Hide resolved
SwanCustomEnvironments/swancustomenvironments/templates/customenvs.html
Outdated
Show resolved
Hide resolved
|
@diocas please have a quick look when you have a moment, thanks! |
SwanCustomEnvironments/swancustomenvironments/scripts/makenv.sh
Outdated
Show resolved
Hide resolved
SwanCustomEnvironments/swancustomenvironments/scripts/makenv.sh
Outdated
Show resolved
Hide resolved
SwanCustomEnvironments/swancustomenvironments/scripts/makenv.sh
Outdated
Show resolved
Hide resolved
SwanCustomEnvironments/swancustomenvironments/scripts/makenv.sh
Outdated
Show resolved
Hide resolved
SwanCustomEnvironments/swancustomenvironments/scripts/makenv.sh
Outdated
Show resolved
Hide resolved
SwanCustomEnvironments/swancustomenvironments/scripts/makenv.sh
Outdated
Show resolved
Hide resolved
SwanCustomEnvironments/swancustomenvironments/scripts/makenv.sh
Outdated
Show resolved
Hide resolved
SwanCustomEnvironments/swancustomenvironments/scripts/makenv.sh
Outdated
Show resolved
Hide resolved
- makenv commented code cleaned - fix on informative comments - custom python option on makenv gets left out
| acc-py venv ${ENV_PATH} | ||
| else | ||
| message="Creating" | ||
| [ -d "${ENV_PATH}" ] && message="Recreating" |
There was a problem hiding this comment.
We don't recreate anymore.
There was a problem hiding this comment.
Actually we do, but not intentionally. Basically, in normal conditions, this is what happens. We got:
spawner -> customenvs extension -> jupyter notebook
When this flow happens (spawner -> customenvs extension (error) -> spawner -> customenvs extension), we don't need to recreate because the whole session is deleted and launched again (from the script perspective, it is its very first time)
However, when this flow happens (spawner -> customenvs extension -> jupyter notebook (go to previous page) -> customenvs extension) the session wasn't even deleted, which means the script knows isn't the first time it is being executed, so it recreates the environment.
This starts to make even more sense with notebook URLs, since you can paste them several times and recreate multiple environments with differente configurations and/or notebook links (not forget they gotta have the same name to be considered a recreation) within the same session.
There was a problem hiding this comment.
This starts to make even more sense with notebook URLs, since you can paste them several times and recreate multiple environments with differente configurations
But we shouldn't allow this, right? We must restrict to just one environment at a time in a SWAN session. And if a custom environment fails to start or the user goes back to the URL that triggers the CustomEnv extension, we ask the user if they would like to try again and if so, we delete their session and they can start another one again. Isn't this safer?
SwanCustomEnvironments/swancustomenvironments/scripts/makenv.sh
Outdated
Show resolved
Hide resolved
SwanCustomEnvironments/swancustomenvironments/scripts/makenv.sh
Outdated
Show resolved
Hide resolved
| [ -d "${ENV_PATH}" ] && message="Recreating" | ||
| echo "${message} virtual environment ${ENV_NAME} using Generic Python..." | ||
|
|
||
| python -m venv ${ENV_PATH} --clear --copies |
There was a problem hiding this comment.
You add the --clear because under some circumstances we might be recreating the environment (e.g. running this script twice)? Is it when the script fails to run and you give the user the option to retry?
|
|
||
| # Install packages in the environment | ||
| echo "Installing packages from ${REQ_PATH}..." | ||
| pip install ipykernel |
There was a problem hiding this comment.
I believe I shared with you some code to find the ipykernel version you need to install here? Which is the same we use in our Jupyter server.
I think it was something like:
pip install ipykernel==`python -c "import ipykernel; print(ipykernel.__version__)"`
There was a problem hiding this comment.
And of course some comment is needed (something like "Install same ipykernel that the Jupyter server uses"). Note that the python we are running in the command to find the version is (should be) the mamba one in the image.
SwanCustomEnvironments/swancustomenvironments/serverextension.py
Outdated
Show resolved
Hide resolved
SwanCustomEnvironments/swancustomenvironments/serverextension.py
Outdated
Show resolved
Hide resolved
SwanCustomEnvironments/swancustomenvironments/serverextension.py
Outdated
Show resolved
Hide resolved
| var project_folder = '/'; | ||
| if (repository.startsWith('http')) { | ||
| // Extract repository name from URL | ||
| const pattern = /^https?:\/\/[^/]+\/([^/\s]+)\/([^/\s]+)/; |
There was a problem hiding this comment.
This pattern and the one of the spawner should be the same no?
There was a problem hiding this comment.
I was a lot more flexible with these regexes cause they are only intended to extract the path for redirecting the user to the correct URL path.
Example: repo = /eos/user/u/username/folder1/folder2/folder3
URL path = /lab/tree/folder1/folder2/folder3
| project_folder = match ? `/tree/SWAN_projects/${match[2]}` : ''; | ||
| } else { | ||
| // Extract CERNBox path for EOS path repository, e.g /eos/user/u/username/project/local -> /tree/project/local | ||
| const pattern = /^\/eos\/user\/([^/]+)\/([^/]+)(\/.*)?$/; |
There was a problem hiding this comment.
Same comment as above, this time for an EOS repo.
Mmm what I don't know is what happens here if the user specifies a non /eos/user path, @diocas ?
There was a problem hiding this comment.
I believe (and hope so) there's no way to redirect the users to a path outside of their home path.
That's why I thought I wouldn't be a good idea to allow a path with just /eos in the spawner.
I suppose the best thing to do was to make sure we got at least 4 elements in the EOS path /eos, /user, initial (/u) and /username.
This way we can be a little more confident that the user is accessing user content (not necessarily his, can be acessing resources of a mate)
There was a problem hiding this comment.
We need to match, for now, /eos/user (here and everywhere we match). Reason: the UI is not ready to access files not on eos user.
SwanCustomEnvironments/swancustomenvironments/templates/customenvs.html
Outdated
Show resolved
Hide resolved
Co-authored-by: Enric Tejedor <enric.tejedor.saavedra@cern.ch>
Co-authored-by: Enric Tejedor <enric.tejedor.saavedra@cern.ch>
Co-authored-by: Enric Tejedor <enric.tejedor.saavedra@cern.ch>
Co-authored-by: Enric Tejedor <enric.tejedor.saavedra@cern.ch>
…envs.html Co-authored-by: Enric Tejedor <enric.tejedor.saavedra@cern.ch>
show back button in case of eventsource errors
| class SwanCustomEnvironmentsApiHandler(APIHandler): | ||
| """API handler for creating custom environments""" | ||
|
|
||
| def initialize(self): |
…object, otherwise the output is not sent to frontend
| else | ||
| message="Creating" | ||
| [ -d "${ENV_PATH}" ] && message="Recreating" | ||
| echo "${message} environment ${ENV_NAME} using Generic Python..." |
There was a problem hiding this comment.
Didn't I leave over here a comment about recreating envs or not? I cannot find it.
Result:
Custom environments can now be built through a web interface.
Developments:
Adopting a SSE approach, the /customenvs URL (with all needed arguments passed on body) requests the python API.
The API gets the provided arguments and executes a bash process (makenv). Its output is streamed and sent progressively back to the frontend, where it gets displayed line by line.
When the script output gets to the end, a specific message (EOF) is sent to inform the client it must stop waiting for messages.
The connection is then closed and the process is finished.
ToDo:
Fix results displaying process
Move /customenvs arguments to body request, instead of url params