GUI for Repobee

[davbeek]

To improve the user-friendliness of Repobee for teachers without a Computer Science background, I would like to extend a subset of Repobee with a GUI: initially just for setting up and cloning student repos. After having discussed this with several others, including Huub de Beer, my ideas are as follows:

The optimal solution would be to develop an LMS app based on an LTI interface. In particular for Canvas, which is the LMS used at our university. Once such an app for Canvas is operational, adapting it for another LMS via LTI would be straightforward.

The problem with this is that an LMS app is tightly integrated with the LMS. In the past, a Canvas app from a commerical software provider has caused Canvas to crash for the whole university, so our IT staff will only approve such an app after heavy testing and quality guarantees, which can take a long time.

An alternative would be to develop a web server which interacts with Canvas via the LTI interface. This has the disadvantage of the required maintenance and user authentication. Teachers will not use something that is not 100% reliable.

Perhaps the best option is to develop a GUI in for instance PySimpleGUI as a wrapper for TKinter and then generate Mac and Windows apps with PyInstaller.

I am currently in the process of defining a SEP project, which is a Software Engineering Project for 10 bachelor students Computer Science who each put in 280 hours. The earliest start date for this is Q3 (February 2022). Other options I am considering:

• Trying to find a teacher assistant or scientific programmer to do the job.
• Hiring a company to do the job. That would not be my first choice and I probably would need a higher budget.

The idea is to fork Repobee to add the required GUI. Everything should be kept open source, including the paid solutions. What are your thoughts on supporting this project?

[slarse]

Hi @davbeek!

Supplying a GUI for RepoBee has long been something I've wanted to do, but have deemed it too much work for me to maintain on my own. I'm all for supporting such a project, though, and I have done some preliminary prototyping on this, however, and have some knowledge to share.

What I've tried

I've explored some options, and come to the conclusion that the only feasible way to maintain a fully featured GUI would be to use a CLI-to-GUI converter, like Gooey or Wooey.

At the time of trying them out (about 2 years ago), Gooey had trouble with nested subparsers (which still seems to be a thing, see chriskiehl/Gooey#574). It also relies on WXPython, which I've had really poor experiences with trying to install on various Linux distros.

Wooey almost worked, but there were some problems with mutually exclusive options (which remains unresolved, see wooey/Wooey#289). For a complete GUI, I think patching Wooey to work with the complex parser that RepoBee employs is the fastest route. What's super cool is that plugins can be supported as well, as the GUI is can be generated on the fly. It's also pretty platform independent as it just uses Django to provide a localhost website.

Both Gooey and Wooey don't appear to have gotten much maintenance in the past year, which could either indicate that they are abandoned, or that they simply work pretty well.

Another thing I've tried is to put a minimal subset of the CLI (I think it was repos setup and repos clone) into a Kotlin application, which just calls the Python app. RepoBee's CLI is stable under semantic versioning, so it's fully feasible to just call the CLI itself; a GUI doesn't need to integrate into RepoBee. This lets one use any programming language. Python isn't precisely the best tool available to build portable GUIs, even though it's gotten a lot better in recent years.

How to go about this

Let's first discuss your proposal: a GUI for a subset of RepoBee. To me, the best way to accomplish that would be to build a GUI that just uses the CLI (there is also a minimal Python API for the core application that can be used). There is no particular need to integrate deeper into RepoBee. The prototype I created long ago (and have subsequently lost) was written in Kotlin, accepted a few inputs for repos setup and repos clone, and then just echoed the stdout output in a "fake" terminal. Was just a couple of hundred of lines of code and worked well enough. It was meant to be "RepoBee with training wheels", but I never had the time to take it to a user friendly state.

The optimal solution would be to develop an LMS app based on an LTI interface.

I'm guessing you meant optimal for your use case? For the general use case it's, as you say, rather limited, and I also have 0 experience of LTI. So I can't provide any expertise in that regard.

Perhaps the best option is to develop a GUI in for instance PySimpleGUI as a wrapper for TKinter and then generate Mac and Windows apps with PyInstaller.

There are two problems with this approach. First, PyInstaller creates something of a sticky licensing situation, as RepoBee is MIT-licensed, but some dependencies (which become statically linked when you use something like PyInstaller) are LGPL. The standard library may also contain GPL-licensed parts on some platforms, such as GNU readline, which would mean that GPL license must be adopted. It's all rather legally complicated.

The second problem is that in packaging it all up with PyInstaller more or less kills the plugin system. If I recall correctly, it is possible to select the desired plugins before-hand and package all of that up, but installing additional plugins was prohibitively difficult. Perhaps a better approach is to provide a mini-application in which you select the desired plugins, and then it packages RepoBee up for the desired platform (or on the current platform). That would potentially also side-step licensing issues: if you package up the software yourselves and distribute it internally, you're free to do so without adopting any kind of license at all.

Windows apps

RepoBee does not officially support Windows, all of our current Windows users just use WSL. Most things should work pretty well, but I know there are problems with temporary directory creation, and there's bound to be assumptions of *NIX-ish environments throughout RepoBee and its plugins. As RepoBee does quite a lot with the file system, natively supporting Windows is not non-trivial, and isn't something I'm keen to pursue myself without significant interest. Of course, I'd accept patches to improve Windows compatibility!

I am currently in the process of defining a SEP project, which is a Software Engineering Project for 10 bachelor students Computer Science who each put in 280 hours.

10 students seems like a lot to throw at this problem. Unless these are senior developers masquerading as bachelor students, my feeling is that coordinating that many students on a problem this small is going to be more trouble than just solving the problem. I think anything more than 2-3 people is too much.

If we're however talking about creating a complete GUI for RepoBee, perhaps by patching and maintaining Wooey, then maybe it would start to make sense. But 10 still sounds like a lot to me, just my gut feeling.

Trying to find a teacher assistant or scientific programmer to do the job.

This is probably the most efficient solution.

Hiring a company to do the job. That would not be my first choice and I probably would need a higher budget.

Yeah, I wouldn't bother with that. An experienced developer could knock out the GUI you're proposing in a couple of days. It's really too small a project to propose to a company.

The idea is to fork Repobee to add the required GUI

I don't see a reason to fork RepoBee. Like I mentioned, I think the easiest way to accomplish this is to just create a small GUI that calls RepoBee's CLI.

So, that was a long post! Take your time to read it through :)

[slarse]

I started writing that long comment, then I went for a walk and came back to it, so it may seem a little bit on the wrong side of ramblings

[davbeek]

Thanks for the long and detailed comment. I really appreciate it. It is a lot to digest, but I am very happy to read that you already have quite a bit of experience in this.

[Alberth289346]

Hai,

I am somewhat involved in this as well, discussing it with @davbeek in the background.

To correct one thing, those 10 students would be learning the agile development process rather than writing software, so the amount of work that can be done there is limited. I agree there isn't much work though, perhaps a study on the GUI screen design, and then see how far it gets towards execution.

I have done some thinking about GUI + repobee too, and it seems to me that you need 2 threads, one for the GUI, and one for repobee since the latter may take time to execute. It is good news that the CLI interface is stable, however you'd still have to deal with the shell escape stuff, and interpreting responses coming back (eg differentiate between info, errors, and progress output). In addition, it's somewhat silly to have 2 Pythons running especially if you want to wrap both into one package in some way eventually. (I wasn't aware of the licensing troubles, but I agree that should be properly covered before going there.)

So I had the idea to divide repobee functionality from its CLI interface, having a Python API in-between.
In other words, the CLI would talk to the API, the API performs the requested function. This can be quite minimal, eg all CLI commands as they are now, right after rescuing the relevant information from the command-line options and arguments.

In the API there should be some way to receive output from eg git, but also feedback from the repobee application, eg errors. Again this could be quite minimal, eg provide a Feedback class instance to the API with a method like

def do_stdout(self, line):
   print(line)

function. In the GUI thread one can the make a derived class, and re-implement the do_stdout method.

I briefly looked in the code to see any was done in that area, but I didn't find it.

I am not sure if the API would need anything else, and absolutely no idea how much work would be involved here.
Given that the CLI is stable, if one sticks to the CLI closely, the CLI <-> API code would be stable too.

[slarse]

Hi @Alberth289346,

To correct one thing, those 10 students would be learning the agile development process rather than writing software, so the amount of work that can be done there is limited.

Right, I suspected it was something like that.

and interpreting responses coming back (eg differentiate between info, errors, and progress output).

Well, with my design you wouldn't need that. The GUI I made was just a thin wrapper around the CLI, with an embedded terminal. Having a tangible link between the GUI and CLI makes the GUI double as a training mechanism for actually using the CLI in the end.

In addition, it's somewhat silly to have 2 Pythons running especially if you want to wrap both into one package in some way eventually

It's suboptimal, but it has two large benefits. First, it's easy. Second, RepoBee is not designed to run in a long-running process. That is to say, there's an assumption that one RepoBee command == one process, which holds perfectly fine for a command-based CLI. I've no idea what kinds of bugs might start to appear if the process that runs RepoBee runs perpetually.

So I had the idea to divide repobee functionality from its CLI interface, having a Python API in-between.

This division exists. First of all, there's the repobee.run function, which is essentially just an API version of the CLI. It's public and stable, and doesn't depend on the CLI.

Then, the CLI itself (see _repobee.cli) is completely disconnected from the command structure (see _repobee.command).

In the API there should be some way to receive output from eg git, but also feedback from the repobee application, eg errors. Again this could be quite minimal, eg provide a Feedback class instance to the API with a method like

All of RepoBee's output goes through the logging functions in repobee_plug.log, with a couple of exceptions like progress bars and interactive menus that go directly to stdout. The logging functions could be hooked to allow for plugin overrides.

[Alberth289346]

Thanks for the feedback, it would be simpler than I assumed, which is great.

So this means we should stick with repobee as a separate process.

While getting some groceries, I wondered about aborting an action. Obviously, currently you can just hit CTL-C although I am far from sure this is a good idea as it may leave a lot of repos in undefined state.

With a GUI you can't kill the process anymore in that way, at least from the keyboard. The GUI could send a SIGTERM or maybe suggest it to quit by sending and catching a SIGHUP in repobee. No idea if the WSL supports such signals.

All of this seems likely to be a future enhancement at best.

[iupNZdsLJM]

As an aside, when I suggested the SEP project to OP, I was thinking about a project where students would develop an web-based GUI for RepoBee with extra educational features like user-name management, group management, statistics and analytics about students' projects, maybe some automated feedback, and the like, plus an LTI integration. Of course, just making a graphical shell for RepoBee is not a sufficient complex project for a team of ten students in their SEP project.

[slarse]

@Alberth289346

So this means we should stick with repobee as a separate process.

Should is a bit strong, perhaps, it was just my initial thought. But, even though RepoBee wasn't designed to run in a long-running process, I don't actually think there should be much trouble. There's very little mutable state in RepoBee, the entire application is mostly functions calling functions and explicitly passing state along. The only place where objects with mutable state comes into play is in plugins, and with repobee.run, plugins are unregistered after each command, causing related objects to go to the GC. So, using one process should actually be feasible.

Do note however that CPython is limited by the Global Interpreter Lock (GIL), and so running two threads doesn't actually mean that you can do parallel computation. For that, you still need two processes. For computationally inexpensive tasks that may still take long (e.g. waiting on the network), asyncio works well and is used heavily in RepoBee, but I don't know how well adopted that is in GUI frameworks at this time.

No idea if the WSL supports such signals.

WSL(2) is a full Linux distribution running under Windows in some form of well-integrated virtual machine (not sure about the details). It should support all signals. What I'm not sure about is actual GUI support, though, it seems to still be under development. For proper GUI support, native Windows support in RepoBee may be necessary, or using a web-based GUI.

@htdebeer-tue

I was thinking about a project where students would develop an web-based GUI for RepoBee with extra educational features like user-name management, group management, statistics and analytics about students' projects, maybe some automated feedback, and the like, plus an LTI integration

Ah, okay I see. I obviously can't speak for the caliber of your students, but from personal experience of working in 10+ groups of bachelor students, the deliverables aren't particularly... usable. That being said, if you want to put a group of students working on something cool related to RepoBee, I'm all for providing guidance and support to them.

All that said, I'm very positive to the idea of a GUI for RepoBee, or even just a subset of RepoBee. In my mind, it would serve its purpose best if it provided a hassle-free experience of working with the basic functionality of RepoBee (primarily setup and clone), to ramp up to using the CLI. RepoBee with training wheels, so to speak. Of course, the ideal situation would be a full GUI for all of RepoBee, but that's too much of a maintenance burden unless we get Wooey working. Which I will have another look at as soon as I can to see how far away it is from working now (it's been years since I tried it last).

[iupNZdsLJM]

Ah, okay I see. I obviously can't speak for the caliber of your students, but from personal experience of working in 10+ groups of bachelor students, the deliverables aren't particularly... usable. That being said, if you want to put a group of students working on something cool related to RepoBee, I'm all for providing guidance and support to them.

Awesome!

By the way, this SEP project counts as the students' bachelor end-project. They operate as a team of software engineers and solve a non-trivial problem for a real client. Two project managers guide their process and a teacher supervises the project. Almost always the clients are happy with the results. The code quality varies, of course, but is satisfactory over-all. Unfortunately, questionable design decisions are not uncommon, and maintainability is often an afterthought at best.

[davbeek]

After having carefully studied all replies, I think I need to clarify my ideas.

First, I think that my target user group is different from the current Repobee user group. My target user group would probably not be using a CLI and they would not be inclined to install the WSL in Windows.

Second, my vision is to create a native Repobee app for the Mac and for Windows. For the Mac in the form of a .dmg, which contains a Repobee.app and for Windows in the form of an .exe install file. The user interface for these two apps would be indentical and would not show any CLI related info.

As a result, Repobee would need to become platform independent (as it is now GitHub/GitLab agnostic), which might be a problem because Repobee was developed specifically for *NIX-ish environments.

I see an LMS app based on an LTI interface as the optimal solution for the TUe teachers and in fact as optimal for all teachers who work at an institution that uses an LMS for their courses, because such an app would not require any tools to be installed and it would not require any additional authentication.

With respect to the SEP project, I should add that for any SEP project it is important that the work can be divided into relatively independent parts. In this case, I would expect the 10 students to divide themselves into sub groups that each work on a different solution in providing a GUI for Repobee. One group could develop an LMS app based on an LTI, another group could focus on making Repobee platform agnostic, a third group could focus on the graphical UI and a fourth group could work on creating apps for the Mac and for Windows. I don't know if it is the client's responsibility to indicate these possiblilties or if the client can just state his wishes regarding the possible forms of the end product and the students figure how to do it.

[slarse]

@htdebeer-tue

By the way, this SEP project counts as the students' bachelor end-project.

Right, so that differs from my experience of large 10+ student projects. When I did my "grand project" with 11 other bachelor students, we were about half-way through a bachelor program. The focus for us was also significantly more on the process than on the product. And the product was... not great (but we got an A, because the specification that the product in no way fulfilled was exquisite).

@davbeek

First, I think that my target user group is different from the current Repobee user group. My target user group would probably not be using a CLI and they would not be inclined to install the WSL in Windows.

Yup, that's clear to me. But there may be a cause for "meeting halfway". That is to say, a GUI that's sufficiently simple to use by the CLI-fearful, but is also useful to those that need onboarding to the CLI experience through a GUI. A thought that comes to mind is to be able to hide/show the CLI when using the GUI.

And for GUI apps in WSL, it doesn't matter, because WSL doesn't yet support GUIs very well.

which might be a problem because Repobee was developed specifically for *NIX-ish environments.

Well, I'd more say it was developed in semi-disregard of Windows. The vast majority of the code base is platform agnostic, there are just a few liberties taken with file system conventions that won't transfer well to Windows. And, of course, the installer is a bash script :)

Supporting native Windows is however a pretty big commitment for RepoBee, because Windows is so different from *NIX systems. I'm not opposed to the idea. Supporting Windows just with the core RepoBee python package probably isn't that much work. But doing it for external plugins, installer, etc is a lot more tricky due to the file system differences. Since I'm still essentially a one-man army, with only the occasional outside contribution, I'm hesitant to commit to supporting Windows without some assurance of more help. I don't want to bite off more than I can chew.

Second, my vision is to create a native Repobee app for the Mac and for Windows.

That'd be great, sure! For Mac, there really isn't much stopping it except for some hiccups with the plugin system (they are solvable), and the aforementioned licensing issues. I've actually created an executable before that worked on Linux and macOS, so it's doable. The licensing problems have me a bit nervous in terms of providing an executable myself, which is why I suggested providing a small application that a semi-tech-savvy user could use to package RepoBee with a select set of plugins.

For Windows, we of course have the native Windows incompatibility to worry about, which is almost entirely file system related.

With respect to the SEP project [...]

That's a pretty grand plan, The thing that I need to make clear is that I can't commit to maintain the output of 10 students' work. I also can't provide any guidance in terms of LTI, as I've never used it, and am very unlikely to use it in the near future.

You know much better than I do how your SEP projects turn out in terms of the actual tangible output. But I think the grander problem is: who will maintain everything once it's done (assuming it's actually usable)? I can guarantee that even if you fork RepoBee such that you can freeze its development, and the students create some truly awesome GUIs based on that, they won't be bug-free. Someone needs to stick around for maintenance. I mean, RepoBee is of course not bug-free as it stands, I need to actively maintain it to keep the bugs at bay!

So if the goal is a usable GUI for RepoBee that stands the test of time, I think less is more.

[davbeek]

Yup, that's clear to me. But there may be a cause for "meeting halfway". That is to say, a GUI that's sufficiently simple to use by the CLI-fearful, but is also useful to those that need onboarding to the CLI experience through a GUI. A thought that comes to mind is to be able to hide/show the CLI when using the GUI.

Sounds good. I think we should go fo a win-win situation, where you and I are both happy about the result.

Well, I'd more say it was developed in semi-disregard of Windows. The vast majority of the code base is platform agnostic, there are just a few liberties taken with file system conventions that won't transfer well to Windows. And, of course, the installer is a bash script :)

That also sounds promising. I think the bash script installer will not be needed anyway for the native Mac/Windows /LMS app.

Since I'm still essentially a one-man army, with only the occasional outside contribution, I'm hesitant to commit to supporting Windows without some assurance of more help. I don't want to bite off more than I can chew.

I completely understand that you need good and proper support for all things related specifically to Windows.

With respect supporting to the SEP project. You would only be expected to give support with respect to the current way of working of Repobee. I think @htdebeer-tue can provide the LTI support.

With respect to the output of the SEP project:

• If we are lucky and have a really good group of students, the results could serve as a basis for a professional developer to finalize some of the useful components and I completely agree that you also need support for maintaining the products and keeping them running and fixing bugs.
• If we have bad luck, somebody will have to redo most of the work.
• In all cases, however, I think we will learn from the results in tems of what works and what doesn't work.
• Also, I would hope to be able to use the results of the SEP project as a show case of what is possible, and, if necessary, for the definition of an additional BOOST project to deliver a professional end product. Every year we have new calls for BOOST projects.
• Concluding: I don't see the SEP project as the solution to our wishes, but as a step in the way to get there.

So if the goal is a usable GUI for RepoBee that stands the test of time, I think less is more.

I hope to be able to realize two products at the same time:

• A usable GUI for RepoBee that stands the test of time, where less is indeed more.
• An LTI app for Canvas, where I have no idea what to expect, but where I am sure that we will learn a lot on the way. For this your support should be minimal.

[slarse]

Sounds good. I think we should go fo a win-win situation, where you and I are both happy about the result.

Agreed.

That also sounds promising. I think the bash script installer will not be needed anyway for the native Mac/Windows /LMS app.

It somewhat depends. Last time I tried packaging everything into a single executable with PyInstaller, there were problems with the plugin system. It may be possible to work around these problems, but it also may not be. It could be the case that a pre-package application can only deliver what's actually packaged into it (so, the one who does the packaging must decide on plugins etc at the time of packaging).

I completely understand that you need good and proper support for all things related specifically to Windows.

That would be nice indeed!

With respect supporting to the SEP project. You would only be expected to give support with respect to the current way of working of Repobee. I think @htdebeer-tue can provide the LTI support.

And that is entirely ok with me. If we're lucky, some of the students may get interested and stick around after the project ends!

I hope to be able to realize two products at the same time:

* A usable GUI for RepoBee that stands the test of time, where less is indeed more.

* An LTI app for Canvas, where I have no idea what to expect, but where I am sure that we will learn a lot on the way. For this your support should be minimal.

That sounds like a feasible plan. Indeed, as neither of us know really what to expect from an LTI app (I barely know what an LTI app is, does it integrate into the LMS or is it standalone and communicates over LTI?), using an SEP project as a testbed for ideas should be mutually beneficial for us and students.

[davbeek]

I am considering posting a position for a TA this week, for the most straightforward GUI version for Repobee. I hope that I can find a suitable person to do the job in a reasonable amount of time. If that works out, The SEP project might then not be needed.

My Boost project at TU/e was evaluated yesterday. They were quite positive, helpful and also very interested in Repobee.

[slarse]

That's great news! I just last week did a quick presentation of a short paper at ITiCSE, and a GUI was one of the things that popped up during discussion, in making the initial adoption of RepoBee, and I quote, "less scary" :)

[davbeek]

The job vacancy has been approved and is now online. It is visible for all students on the tue.jobteaser.com career center, so not for other people. It will stay online until 31-12-2021, because I have no idea what to expect in terms of number of useful candidates. Will let you know when I think I may have found a suitable candidate.

[slarse]

Hi @davbeek,

I've been giving the whole GUI thing some more thought lately, and it does seem like the next logical step in making RepoBee easier to adopt. Really, I think it must happen for RepoBee to get adopted by any but the most adventurous teachers. And in that, I think the end goal should be a GUI that to a great extent has feature parity with the CLI.

Now, a first stab at this of course doesn't have to live up to that, but ideally I think it should be built in such a way that the end goal of CLI/GUI feature parity is feasible. I see a couple of approaches here.

1. Generate the GUI from the CLI. This is the approach taken by tools like Gooey and Wooey (see GUI for Repobee #912 (comment)). The major downside of this is that it requires some fairly finicky introspection of the CLI, which is made to work for most use cases (but doesn't work flawlessly for all).
2. Abstract the user interface to something less concrete than either GUI or CLI, and then generate both from that abstraction. This is sort of what's already going on with adding command line options in RepoBee's plugin system. That would just have to be further generalized.

As I've tried both Gooey and Wooey, and hit roadblocks with both, I'm thinking option 2 is the best way to go. I'm thinking that a generalized specification of a user interface could be provided by a developer, and then RepoBee generates a GUI and CLI from it.

Currently, the core parts of the CLI are hand-crafted, while plugins are automatically generated and integrated into the core CLI at runtime. This is more due to legacy reasons than there being some technical reason why the entire thing can't just be generated from more abstract specifications. I've been wanting to remove the hand-crafted parts of the CLI for some time as they're kind of a paint to maintain, and abstracting everything one step further to allow for other kinds of user interfaces. One such interface would be the CLI, and another could be a GUI created with PySimpleGUI.

RepoBee is almost exclusively designed to be a "command-based" tool, i.e. the user selects a command to run, gives it a bunch of inputs, and then RepoBee does its thing without more user interaction (there are a few notable exceptions, like the plugin manager and the config wizard). This means that it's not technically speaking that hard to generate the UI for it based on some abstract specification. More importantly, that abstract specification doesn't need to be all that involved, it just needs to be able to express a set of commands, each with different sets of valid inputs.

One big but is user feedback. GUI user feedback and CLI user feedback are inherently different. Currently, RepoBee just prints stuff to stdout. The path of least resistance would be to just show those printouts in the GUI, in some reasonable fashion.

Interactive commands, like the plugin manager, are obviously a lot harder to generalize in this way. However, those commands don't make much sense for a GUI anyway, and would be better replaced with just a form.

I just wanted to share some of these thoughts. My ambition is to start tinkering with some prototyping in the coming weeks for abstracting and auto-generating the entirety of the core CLI, alongside some proof of concept of GUI generation.

[davbeek]

I am late for a grading deadline for my bachelor students, so will come back to you on this later!

[slarse]

No worries, I probably won't get around to this until the Christmas holidays. And it was mostly meant as an FYI for what my thoughts are on the whole GUI thing: I think it should be the focus for future efforts. If my entire next year of working on RepoBee is spent only on facilitating and/or helping out creating a usable and maintainable (that's key) GUI, then that'll be time well spent.

[davbeek]

@slarse I have sent you an elaborate email explaining the current state of the repobee gui developments. Hope you have properly received it.