Easy updating of software titles & update notifications

[din14970]

After multiple years of running DietPi on my Raspberry Pi 4, I (shamefully) only recently learned that updating of the software itself is not handled automatically. I needed to update NextCloud all the way from v22 to v25 (last version supported on armv7), which was quite a pain. Indeed after looking into the Dietpi code (dietpi-software), it seems an install always tries to install the latest version of a title, and each software title has its own update instructions. Confusingly, updating some software requires re-installation, which can completely mess up custom configuration.

The feature I felt like I was missing was a universal update command/menu item for software titles, and some notification on log-in which would alert me when new versions of installed software are available (similar to what now exists for apt packages). Ideally, running update would also give a description of what would change in the system, so that people know when their custom configuration will be erased.

For updating, I envision this would entail adding another function for each title, next to install and uninstall, called update. For notifications I guess this would also require a function per title to get the latest version tag + a cronjob to pull these tags and compare them to the current versions.

It's just an idea and I guess it would involve quite a lot of work... I searched discussions and issues to see whether this idea was mentioned before but could not find it. Nevertheless, if it is a duplicate feel free to close.

[MichaIng]

Thanks for your request.

Indeed, while we do upgrade some software titles where we build and host own packages for, or when upgrades are important for bug fixing or security reasons, but otherwise they need to be done manually, especially when software titles have internal updaters available via their own interfaces, like in case of Nextcloud.

What is always upgraded are APT packages, but this covers only a small part of what dietpi-software offers. APT packages are trivial to install, while we focus on software which is more complicated to install and/or setup.

The problem is that every software (which is not just installed as APT/DEB package) needs to be updated differently. dietpi-software reinstall ID can be used to reinstall software, which often re-downloads or re-compiles binaries, hence implies an update, and preserves data and configs. Hence it often is the intended way to update software, also according to our docs in those cases: https://dietpi.com/docs/software/cloud/#__tabbed_7_6
But e.g. in case of Nextcloud, one cannot upgrade across multiple major versions, but v22 => v23 => v24 => v25 ... in multiple steps, as you recognised, which is hard to evaluate/do correct in a script like ours. Also their internal updater creates backups, does file integrity checks and other reasonable things, which we wouldn't be able to reasonably maintain in our script. While in case of Nextcloud, required database/config migrations are done automatically in a dedicated step, after files have been replaced, in other cases internal updates can imply mandatory steps prior or along with replacing the files, which we cannot know or do in our script. For this reason, in most cases where internal updaters exist, dietpi-software reinstall ID won't update the files, but instead print an information that the internal updater must be used, and if this is not possible, as something is broken with the instance, the installation directory must be removed manually for dietpi-software to re-download/install (the latest version).

Those software titles which are intended to be updated via dietpi-software reinstall (according to our docs), should all preserve custom configs and data. There are rare cases where one setup steps requires another one, which can change a particular setting, but usually there is then also a backup or the old config generated generated. What we do replace on reinstalls is own config files in conf.d-like directories with dietpi- prefix. These usually contain the mandatory bits to make it work together with our service unit or other parts of the setup, and it is intended to create own config files in conf.d with higher alphanumerical order, to override individual settings of the dietpi- config. Those will then survive a reinstall.

So essentially, dietpi-software reinstall ID does an update in those cases where we think it is safe and sufficiently simple. It implies some basic setup steps, but preserves custom configs and data in most cases, especially those where it is the intended way of updating, else keeps at least backups. But it can be made more visual in the GUI, where it is currently not present at all.

I agree that it would be great to have more details/an overview of what dietpi-software reinstall does in which case, i.e. whether it updates the software itself, or just dependencies/DietPi-specific configs, which files are replaced, which ones are edited, which ones remain untouched. As a start we could add this step by step to our documentation. I anyway planned to implement a reinstall menu in the dietpi-software GUI, but it requires some kind of system/structure to add mentioned information there, and would still mean a lot of code. One of our contributors is working on splitting the script in multiple directories and scripts for each software title. With e.g. a dedicated file which contains software meta data (name, category, dependencies, docs URL, ...), it becomes easier to add more metadata without massively increasing the size of an already huge single script. I could think of something like this about what a reinstall:

• Replaced files/directories, backups generated or not
• Files which are edited, in case which exact config keys (probably too much)
• Files/dirs which are kept untouched
• Whether it always updates the core software version, or whether their internal updater should be used for this. But as a general matter of fact: Using an internal updater is always the preferred way and should always work best, regardless what a reinstall does.

So bear with us that it will take some time, but I am aware of the lack of prominence and information around reinstalls. Until then, it would be great if you could inform us, if a particular reinstall leads to lost custom configs, or edits some config too aggressively. I did a lot of changes the last year to make reinstalls less intrusive and more careful, and aim to continue doing so, if needed.

[din14970]

The problem is that every software (which is not just installed as APT/DEB package) needs to be updated differently.

Right, just like each software title requires a different install and uninstall procedure, which is nicely hidden by Dietpi behind a uniform interface. I propose a similar thing for updates. But indeed, this will require a custom implementation for almost all software titles.

For most software, dietpi software update could be an alias for dietpi reinstall. But for Nextcloud, the dietpi update could just wrap the internal update + DB migrations, and do this in a loop until the latest version is achieved. But I can imagine that writing all this logic in bash is totally not trivial or quick, especially in the current code structure.

What we do replace on reinstalls is own config files in conf.d-like directories with dietpi- prefix. These usually contain the mandatory bits to make it work together with our service unit or other parts of the setup, and it is intended to create own config files in conf.d with higher alphanumerical order, to override individual settings of the dietpi- config. Those will then survive a reinstall.

This is good information, thanks!

One of our contributors is working on splitting the script in multiple directories and scripts for each software title. With e.g. a dedicated file which contains software meta data (name, category, dependencies, docs URL, ...), it becomes easier to add more metadata without massively increasing the size of an already huge single script

This would be very helpful, also for adding new software titles which I have attempted but failed to do.

I could think of something like this about what a reinstall

Another one is the creation/deletion of a user. Updating gitea broke my config because of what is described in this thread: https://dietpi.com/forum/t/troubles-sshing-to-gitea/14696/19. I access my repos through SSH but I had to manually modify the /etc/passwd file to do so. Reinstalling gitea undid this change.

So bear with us that it will take some time, but I am aware of the lack of prominence and information around reinstalls. Until then, it would be great if you could inform us, if a particular reinstall leads to lost custom configs, or edits some config too aggressively.

Will do. If I would have time I would be able to contribute to the code, but at this moment I simply don't have the time. As an alternative, I was thinking to start sponsoring instead, but I noticed you only have a link to Patreon. Would you also consider enabling sponsorship directly through Github?

[MichaIng]

Right, just like each software title requires a different install and uninstall procedure, which is nicely hidden by Dietpi behind a uniform interface. I propose a similar thing for updates. But indeed, this will require a custom implementation for almost all software titles.

Basically "reinstall" is this, or aims to be this, for most cases. Calling software specific APIs to trigger their internal updater, or even mimicking their internal updater, is not something we can reasonably maintain. For Nextcloud in particular I know the CLI, but for most others not, and am not even sure whether it exists.

This would be very helpful, also for adding new software titles which I have attempted but failed to do.

Indeed, one reason I had this in mind for was indeed that adding new software options should be dynamically possible by just adding a new dir with expected scripts, which also makes testing of new software implementations much easier, compared to editing a huge script.

Another one is the creation/deletion of a user. Updating gitea broke my config because of what is described in this thread: https://dietpi.com/forum/t/troubles-sshing-to-gitea/14696/19. I access my repos through SSH but I had to manually modify the /etc/passwd file to do so. Reinstalling gitea undid this change.

I hope you do not edit it manually, use usermod or alike? 😉

How exactly did you need to change the gitea user's config to enable SSH? It should be possible OOTB, when using the /mnt/dietpi_userdata/gitea/.ssh dir for keys in case. I remember we did some related change for this 1-2 years ago.

Would you also consider enabling sponsorship directly through Github?

So far I hesitate to split it into too many different donation options aside of PayPal and Patreon. But I'll reconsider, at least of there appear to be more who would donate through GitHub Sponsors program but not through PayPal and Patreon.

[din14970]

not something we can reasonably maintain

pitty, but completely understandable.

How exactly did you need to change the gitea user's config to enable SSH?

As described in that thread, I had to give the gitea user shell access by changing /usr/sbin/nologin to /bin/bash. Otherwise, any pull or push operation gives the error fatal: protocol error: bad line length character: This. If there is an alternative solution described somewhere would be great.

So far I hesitate to split it into too many different donation options aside of PayPal and Patreon. But I'll reconsider, at least of there appear to be more who would donate through GitHub Sponsors program but not through PayPal and Patreon.

Perhaps you can launch a poll. I personally have bad experiences with PayPal; I've never made a Patreon account. Through Github I already sponsor some other projects so it would be quite easy to add another.

[MichaIng]

The gitea and gogs users is now given /bin/dash as default shell OOTB (on installs and reinstalls). It is a little faster than bash, and skips the DietPi login scripts and banner on interactive shells. Not sure whether this is the case with Git actions via SSH, but dash should (bourne shell) not have any downsides: 0dac52b