Replies: 17 comments 26 replies
-
|
Hi, I strongly support pushing forward development. Although I'm not a software developer I'll try to put my two cents in. What are three cool things you have done with nbgrader lately?
What are three problems you have had lately?
Would you want to have an online meet-up? Jens |
Beta Was this translation helpful? Give feedback.
-
|
Hi! I am on vacations, so quick answer for now. Thank you so much @rkdarst for initiating the discussion. By large, I am in line with all proposed items. Four cool things happening here these days: Weak points of nbgrader that are itching us these days (not new though) Man power: Suggestion of action:
@jeflem: looking forward learning more about jupyter-nbgrader-lti integration Cheers, |
Beta Was this translation helpful? Give feedback.
-
|
Yes, we are still active users of NBGrader. Sadly @BertR moved back to mainland Europe, and is no longer part of our team - hence the lack of input from him of late. We are also suffering a shortage of developer time to help improve the code [which I'm vocal about at work], however we are definitely "would if we could" camp. One if the big challenges here is ensuring that what we add to the core product is generally useful rather that I need - a case in point would be LTI integration [I know about this one, as I've been playing with it in other arenas]: I could write an integration that connects Moodle to named assignments - which may well solve my the use-case, however misses a few issues:
The grade passback is definitely something we [in Noteable] are going to be looking at, and I will definitely be encouraging something that does not send us down a "fork of nbgrader" route!! [and I should really try to make more time to spend attention here.... helping & supporting, if nothing else - that's on me] |
Beta Was this translation helpful? Give feedback.
-
|
... and some things I've been wanting to improve, which I thing are generally useful:
I can already see challenges here, when systems use different exchange plugins. |
Beta Was this translation helpful? Give feedback.
-
|
Hi All 👋🏻 Thanks for opening this discussion. I've been wanting to contribute more, but haven't felt aligned with the greater direction of the project. At the @umsi-mads program, we use nbgrader, but not the entire thing. We have a partnership with Coursera and use nbgrader in a cloud context where the exchange just doesn't make sense. I think this cloud use case is worth considering because it's used in-house at Coursera and Vocareum and probably many others, but that might be worthy of an entire fork. I can expand on this if anyone's interested. My cool thing is that to work in this cloud context, I built my own grader that runs as a lambda (still private, sorry) that orchestrates No recent problems, but in the past couple weeks, I've gotten 2 different requests for help in accessing the contents of a Markdown cell from an autograder cell and the solution is really clunky, so there might be something there. I'm more than happy to contribute upstream on nbgrader or anything down through ipython, but I don't have much in the way of direction at the moment. |
Beta Was this translation helpful? Give feedback.
-
|
Can I confirm that you've essentially written an external exchange service [the instructors release to some external store, and students pull from the external store] - like ngshare (other tools are available) I think it would be interesting to collect & collate these examples [we use nbexchange .... there must be more out there - there's plenty people use nbgrader at scale] |
Beta Was this translation helpful? Give feedback.
-
|
Hi, thank you for starting this discussion, I'm pretty new to nbgrader but I would like to get involved and help with future development. Instead of the three cool things we have done with nbgrader lately, I will describe our use case how we started with nbgrader at our university. So this will be nothing new for nbgrader, but it is cool for us 😉 The major problems we faced during the semester are:
In 2024, I would like to get involved in nbgrader development and work primarily on fixing and testing these issues. In the end, I should even get some funding from my university... I think that I have sufficient skills for hacking on a Python/JavaScript project, but as others mentioned, it is difficult to get started so I would like to participate in any meetup or hacking session. Finally, I think this is a good place to mention a few things that I think should be addressed to improve the maintainability of nbgrader in the future:
|
Beta Was this translation helpful? Give feedback.
-
|
Answering my own questions: We run nbgrader as a service for computer science (more like the "data science" part), and also for others in our university. It's running in Kubernetes but uses nbgrader also, and is well-integrated to the rest of our university IT systems, which gives us many nice features but also more complexity. We support courses with a significant fraction of grades in the department. Not every course (perhaps not even many) use autograding because of the difficulties in doing it well. Three cool things:
Three problems:
Many of these are specific to our setup. Overall, teachers really appreciate nbgrader but also understand its limits. |
Beta Was this translation helpful? Give feedback.
-
This is an interesting concept, which I remember floating around before: external autograding services. [currently, we do it by keeping the autograding process in the instructors notebook-server, but have the files in a separate file-store (to bypass quota issues... and, oh boy... did we need to enforce quota issues: it doesn't take long for an enthusiastic power-user to squirrel away gigabytes of data & sample notebook-documents!] |
Beta Was this translation helpful? Give feedback.
-
|
What we will do for now is: update our own repo's main branch (https://github.com/AaltoSciComp/nbgrader/) with pull requests from The idea is this would be more up to date code that fixes bugs, is synced with jupyter/nbgrader:main, and eventually becomes jupyter/nbgrader:main. However... it would be much better to work directly in jupyter/nbgrader:main, so @brichet, I'd say it's up to you: let us know what we should do. If we can somehow help keep upstream updated faster, then by all means we should work there and not try to make alternative main branches. Some of the pull requests we have submitted have been in our production for a while now, someone will comment on them separately. Any thoughts on what we should do next? Or how people can help? For example we deploy PRs in our production and tell if they work? |
Beta Was this translation helpful? Give feedback.
-
|
I think one of the problems is we don't have many people who can actual action PRs [quite a few have moved onto other companies/services/jobs] ..... but yes, trying to get together some form of co-ordinated management would be good |
Beta Was this translation helpful? Give feedback.
-
|
Thanks @rkdarst for opening this discussion. I agree that there is a lack of maintenance/developments in nbgrader. I'm probably the only one who's been merging PRs for a year. But unfortunately I can't devote too much time to it at the moment.
I don't think it's up to me to decide on the future of nbgrader. As an open source project, it belongs to anyone who wants to get involved, and of course mainly to the users (of which I'm not one).
The project is indeed missing some developers involved in reviewing/merging, but I don't think we should merge changes without review (if this is what you mean there). Of course, I advocate keeping the upstream updated with fixes and improvements. A minimal way to review PRs can also be to provide feedback that they don't break anything (and works) in other production deployment. |
Beta Was this translation helpful? Give feedback.
-
I don't mean to merge entirely without review: there should be some balance, for example for big things, if at least someone has run it in production, that's a good sign. But the second two sentences are exactly what I see as the problem: first off, things can become confusing if no one has the overall view. Second, almost any probably is probably depended on by someone (relevant xkcd of course...), and even then one solution may be wrong for someone else. Does anyone have a solution to this? This is what prompted me to make my proposal: it seems we won't have anyone to make the hard decisions. Best (I think) we can do (in the near term) is be more active in accepting things, if
... and hope that by doing the above, we can nurture some people that do have the big-picture perspective in the future. |
Beta Was this translation helpful? Give feedback.
-
|
@rkdarst I've edited your first comment to add missing people who worked on nbgrader last year or were involved in the maintainer discussion. |
Beta Was this translation helpful? Give feedback.
-
|
Hello everyone, Many thanks to @rkdarst for starting this conversation. Seeing so much engagement in this conversation is really inspiring! The work of @brichet at @QuantStack (which was partly funded by @nthiery and @pierrepo) was first focused on porting NbGrader to JupyterLab. Indeed, this was a critical requirements in advancing the Notebook 7 plan outlined in JEP #79. Indeed, the JEP included a commitment to make the most popular classic notebook extensions compatible with JupyterLab - with an explicit mention of nbgrader. However, it's evident that the funding allocated to the project falls short of providing the necessary attention it requires. Despite the fact that the problem of grading notebook as received much less attention that the rest of the Jupyter stack, I am convinced that the issue addressed by nbgrader is important. Clearly, the amount of engagement in this thread shows that there is significant interest in the tool. Several people have invested considerable effort in adapting the current version to their specific use cases. Perhaps, we ought to adopt a more ambitious stance on this matter. For this reason, last fall, we submitted a letter of intent to the CZI to secure funding for nbgrader's development. Unfortunately, it didn't progress to the stage of submitting a full application despite our efforts. The proposal included a significant overhaul of the UI, and work on consolidating the backend. I have included the body of the letter of intent below, which had to adhere to a strict 500-word limit:
I think that it would be interesting to push this kind of proposal forward with another funding source. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for kicking off this discussion @rkdarst ! Context / Usage of NbGrader: I teach a large undergraduate data science course (1,000+ students per semester). We use NbGrader to autograde Jupyter notebook assignments. We use NbGrader only for its auto/manual grading functionality (via Formgrader), and feedback form generation. We don't use anything related to the exchange; I have written other software that handles distributing assignments, collecting them, monitoring TA grading, uploading grades to Canvas, etc. From the students' perspective, they just click a link on Canvas, and they're taken to a browser tab with a Jupyter notebook where they can work. When they're done working, they simply save the notebook and quit. In the background, we're using LTI Authentication into a JupyterHub we run in our department, which spawns a Docker container for the student to work in. At assignment deadlines, we take a ZFS snapshot of our student directories and use those for grading later on. Cool thing: We have quite a few assignments in our course, each with quite a few questions. When we originally designed the course, we manually wrote test code for all the questions, which amounts to thousands of lines of code to maintain and debug. I recently pushed some changes to NbGrader #1817 (also see the new documentation item here) that give instructors the ability to automatically generate test code for objects they want to test. If there's an object named in the Problems: We share a lot of the same problems as mentioned by @lahwaacz in #1862 (comment) . Mostly related to students accidentally breaking notebooks, opaque autograding error messages, etc. But overall I find NbGrader isn't the main source of issues in my class any more -- it mostly does what it's supposed to do. Development: I don't have a tonne of extra capacity for more development here at this point --- I've got other software projects that are in a much worse state that need my attention first ;-) A random thought: I know that a lot of my colleagues are now using PrairieLearn. I don't know much about it, but I wonder if there are features/ideas there that we could map over to NbGrader. |
Beta Was this translation helpful? Give feedback.
-
|
Hello everyone, Thank you @rkdarst for kickstarting the discussion! Since 2019, we have been using nbgrader for exams and assignments at Hochschule Bonn-Rhein-Sieg. If anyone is interested, we presented parts of our work at JupyterCon 2023. You can find the presentation here. What have we done with nbgrader?
What are points were we see room for improvement?
Cheers, |
Beta Was this translation helpful? Give feedback.
-
|
Hi @tmetzl, For information: I'll possibly spend a good chunk of July at the Hausdorff institute in Bonn. Maybe that can |
Beta Was this translation helpful? Give feedback.
-
|
That would be great. |
Beta Was this translation helpful? Give feedback.
-
|
Hello everyone, I'm really happy that this discussion is happening, and that many people are contributing with their experiences using NBgrader. In November 2023 with the help of @SylvainCorlay and @brichet, I started doing some UX research and design on this subject, so I would be happy to contribute further, gather inputs and work with anyone here interested in improving the UX and UI of this tool. I have a UX background and not a technical one, so it's also important for me to gather technical constraints and priorities so that I can take them into account when making design decisions. For the moment I have
I would like to organise or participate in meetings with users and developers to work together on this. I think that the first step would be to meet with all of you who use NBgrader (and/or contribute to its development, but that's a plus), share the state of the UX work and review the user journeys together to be sure I haven't missed any important pieces of the way you use, and need to use this tool. So feel free to reach out if you are interested. |
Beta Was this translation helpful? Give feedback.
-
|
Hello All, would some of you be interested in participating in a UX/project planning online workshop next week? Please put a thumbs up if interested, and I will suggest time slots that could suit all time zones. |
Beta Was this translation helpful? Give feedback.
-
|
@GabrielaVives I would also like to participate if possible. Preferably Monday, Tuesday or Friday. But if it does not work I'm also okay with sitting this one out. |
Beta Was this translation helpful? Give feedback.
-
|
How about this Tuesday (19th) or next Tuesday (26th) from 2 to 4pm paris time? |
Beta Was this translation helpful? Give feedback.
-
|
Both are good for me... |
Beta Was this translation helpful? Give feedback.
-
|
I'm in US/Eastern (currently UTC-04:00). I want to loosely say I can attend those days+times. I'm not really a morning person 😅 |
Beta Was this translation helpful? Give feedback.
-
|
Ok well let's plan this for Tuesday the 26th, to let everyone time to see this and block their calendar. I'll follow this with an invite to the meeting. |
Beta Was this translation helpful? Give feedback.
-
|
Sorry for the very late reply. First, permission setting of root directory for file exchange directories is "777". Second, we fixed nbgrader's codes in order to user "JST" (Asia/Tokyo) timezone. Third, it is difficult for many classes to use the system at the same time. Regards, |
Beta Was this translation helpful? Give feedback.
-
|
Thank you very much @tomoya-saito, sounds interesting but, as you mentioned, quite complex. Me and my group applied for some funding to further develop Ananke and to contribute to related projects (jhub, nbgrader,...). Unfortunately, request wasn't successful. So there's little time for working on things at the moment. Our long-term plan is to move all the LTI/LMS things in Ananke to JHub to have a general LTI interface/API within JHub. Then extend nbgrader to use this API (and connect another tool to it, too). Maybe we should start some discussion on how such a more general LTI API could/should look like. |
Beta Was this translation helpful? Give feedback.
-
|
Yep - a new thread for this would be good...... Noteable only uses LTI for access..... however our's is an even more complicated service :) |
Beta Was this translation helpful? Give feedback.
-
|
Thank you, @jeflem A complex mechanism is needed to solve this permission problem while ensuring user compatibility, and adding to it the concept of "free use of the system by any subject, any user at any time." If it's just a "777" permission issue, it might be resolved within the scope of the "nbgrader" issue. Regards, |
Beta Was this translation helpful? Give feedback.
-
|
Gentles..... can we either move the discussion of the exchange and it's mechanism to a new [not talking about UI stuff] topic, or agree it's an issue that needs looking at [not in a discussion about UI stuff] |
Beta Was this translation helpful? Give feedback.
-
|
I agree we should move this discussion but I quickly wanted to add something that might be of interest. @tomoya-saito @jeflem |
Beta Was this translation helpful? Give feedback.
-
|
Hi Everyone, Here is the Jitsi link to the UX workshop happening tomorrow Tuesday 26th from 2pm to 4pm Paris time. And the calendar event if you wish to add it to your calendar. See you tomorrow! |
Beta Was this translation helpful? Give feedback.
-
|
Many thanks.... See you tomorrow! |
Beta Was this translation helpful? Give feedback.
-
|
Dang-it..... did I miss it? |
Beta Was this translation helpful? Give feedback.
-
Hi everyone,
[This is a survey to see about further nbgrader development: who is interested? Shall we try to do more? Even if you aren't interested, can you say three cool things you have done with nbgrader lately and three recent problems?]
We (Aalto University CS department) have been thinking about trying to push forward with more nbgrader development, at least a little bit. Clearly, there is some development in people's different branches (we've got some), but upstream dev is a bit slow. We understand that and are wondering what we can do about it. Is anyone interested in taking risks and trying to push things further?
We think one problem is that few of us are long-time developers in nbgrader and don't want to risk breaking things for others, or know what might break, or take responsibility if it does, and things mostly work anyway. Thus, development is very slow. What about accepting that things might get unstable and merge pull requests with less threshold? We could declare [0.9?] the stable version recommended for non-experts, and then make the main branch more unstable, where we work rapidly and learn as we go. By working faster, we learn more, build more community, and hopefully can fix things faster and eventually get to where we can .
(the null option: continue as it is, basically, and development is slow)
Does anyone have stats on global nbgrader usage? Should we make a survey to see what's going on in the world?
We (Aalto University CS department) have people hired who could work on this. We can't promise things forever in the future, but have at least one person who can watch over, test things, and try to merge things. Of course, it needs a big community and talking to be able to manage this.
What do you all think?
Could anyone interested in Nbgrader reply with your interests?
This is just one of our ideas, if others have an idea for faster development, we can help with that, too.
I propose a starting point for those interested: can you tell three cool things they have done with nbgrader lately, and three problems?
Richard on behalf of my team
cc people with recent nbgrader development: @brichet @danielmaitre @shreve @jeflem @ykazakov @lahwaacz @UBC-DSCI @perllaghu @trevorcampbell @tuncbkose @tmetzl
Or who replied to the last maintainer team issue #1637: @BertR @nthiery @willingc @damianavila @choldgraf @ocordes @jgwerner @jhamrick @SylvainCorlay @jtpio
Beta Was this translation helpful? Give feedback.
All reactions