Jupyter-Notebook crashes after adding an assignment.

[cablesky]

Hello!

I'm using nbgrader on the following infrastructure:

Ubuntu 20.4
JupyterHub 4.1 in a Docker Swarm Cluster
The directories in the JupyterNotebook are provided via NFS shares from an NFS server running in a Docker container.
JupyterNotebooks Version 4.1
The Issue:
As soon as I add a new assignment in Formgrader, the Jupyter Notebook restarts.

However, this only happens when the Jupyter Notebook is running on a Docker Swarm slave node. If the Jupyter Notebook is running on the master of the Docker Swarm cluster, where the NFS server is also running, the assignment can be created without any issues.

Does anyone have an idea of what happens when an assignment is created, and why the notebook crashes?

[brichet]

As far as I know creating an assignment only creates a new empty directory, and write in the database.
Maybe the issue come from the database writing.

Don't you have some error logs server side when it crashes ?

[cablesky]

Thanks for your answer!
Yes, the problem is generating the database! I just don't know what problem nbgrader has.
A gradebook.db is created, but it has 0 bytes...

The log only says:

[I 2023-12-12 15:10:42.943 SingleUserLabApp list:221] Released assignments:
[I 2023-12-12 15:10:42.943 SingleUserLabApp list:221] Released assignments:
[D 2023-12-12 15:10:42.943 SingleUserLabApp log:191] 304 GET /user/userx/formgrader/api/assignments (userx@137.250.198.90) 5.85ms
[D 2023-12-12 15:10:43.539 SingleUserLabApp auth:1084] Allowing Hub user userx based on oauth scopes {'access:servers!server=userx/'}
[I 2023-12-12 15:10:43.541 SingleUserLabApp log:191] 200 GET /user/userx/api/contents?content=1&1702393843524 (userx@137.250.198.90) 2.58ms
[D 2023-12-12 15:10:47.469 SingleUserLabApp auth:676] No user identified
[I 2023-12-12 15:10:47.469 SingleUserLabApp log:191] 200 GET /user/userx/api (@10.0.21.11) 1.06ms
[D 2023-12-12 15:10:47.612 SingleUserLabApp auth:1084] Allowing Hub user userx based on oauth scopes {'access:servers!server=userx/'}
[I 2023-12-12 15:10:47.616 SingleUserLabApp log:191] 200 GET /user/userx/api/metrics/v1?1702393847590 (userx@137.250.198.90) 3.74ms
[D 2023-12-12 15:10:47.768 SingleUserLabApp auth:1084] Allowing Hub user userx based on oauth scopes {'access:servers!server=userx/'}
[I 2023-12-12 15:10:47.771 SingleUserLabApp log:191] 200 GET /user/userx/api/metrics/v1?1702393847755 (userx@137.250.198.90) 3.95ms
[D 2023-12-12 15:10:48.918 SingleUserLabApp auth:1084] Allowing Hub user userx based on oauth scopes {'access:servers!server=userx/'}
[D 2023-12-12 15:10:48.919 SingleUserLabApp base:37] creating gradebook

Not more... then the Jupyter Notebook freezes, and restart.

and the problem only occurs when the Jupyter notebook is running on a different Docker Swarm node.

[cablesky]

I found out something else interesting:
I have the command:
nbgrader quickstart test-nbgrader
executed.
This takes a while, but the command runs successfully:

nbgrader quickstart test nbg
[QuickStartApp | WARNING] Config option `kernel_spec_manager_class` not recognized by `QuickStartApp`.
[QuickStartApp | INFO] Creating directory '/home/userx/work/test-nbg'...
[QuickStartApp | INFO] Copying example from the user guide...
[QuickStartApp | INFO] Generating example config file...
[QuickStartApp | INFO] Done! The course files are located in '/home/userx/work/test-nbg'.
    
     To get started, you can edit the source notebooks located in:
    
         /home/userx/work/test-nbg/source/ps1
    
     Once you have edited them to your satisfaction, you can create
     the student version by running `nbgrader generate_assignment ps1` from the
     '/home/userx/work/test-nbg' directory.
    
     For further details, please see the full nbgrader documentation at:
    
         https://nbgrader.readthedocs.io/en/stable/
    

The strange thing is that from this point on, creating a new assignment in the form grader works again. The database gradebook.db no longer has 0 bytes and everything works normally.

What does the “nbgrader quickstart” command do to get the database working again?

[brichet]

It is supposed to do the same, I think.

Can't it be related to rights ?
Is this the same user which create the database and directory in both cases (quickstart and formgrader) ?

[cablesky]

Thanks for your answer!
The permissions are the same.
For example, if I start a new Jupyter notebook and immediately click on "Formgrader->Create Assignment", choose a name and click on "Create", the Jupyter notebook crashes.
The resulting "gradebook.db" with 0 bytes has the same permissions as the "gradebook.db" created with the "nbgrader quickstart test" command.

You can also just run nbgrader db assignment add test instead of nbgrader quickstart test - and from this point on, creating assignments via the form grader will work again.

Do you have any ideas as to what this could be?

[brichet]

I believe that with both methods the database is created at

nbgrader/nbgrader/api.py

Line 1327 in 12136c2

db_exists = len(inspect(self.engine).get_table_names()) > 0

And the tables should be created at

nbgrader/nbgrader/api.py

Lines 1331 to 1335 in 12136c2

	if not db_exists:
	alembic_version = get_alembic_version()
	self.db.execute(text("CREATE TABLE alembic_version (version_num VARCHAR(32) NOT NULL);"))
	self.db.execute(text("INSERT INTO alembic_version (version_num) VALUES ('{}');".format(alembic_version)))
	self.db.commit()

Can you copy the output of the command pip freeze (in the environment in which is running jupyterlab) ?