Upgrade mechanism

[kaikreuzer]

We need to have a stable way to upgrade from release to release as well as from snapshot to snapshot.
First this needs to be solved for the distro, then for apt, docker, spk, etc.

My idea for the upgrade would be the following:

• completely replace the runtime folder by the new version
• delete userdata/cache and userdata/tmp folders
• keep conf and the rest of userdata fully in place

With this, I would assume Karaf to reconstruct the OSGi framework with all the latest versions of the bundles and using the new "openhab-base", while using all statically (conf) and dynamically (userdata) created data as before.

[kaikreuzer]

I did some analysis here. A first major problem is #315 - this causes configuration to be lost.

[kaikreuzer]

With #318, the configuration is moved to userdata/etc, so if the upgrade mechanism leaves this untouched, the "lost configuration issue" should be solved with it.

Next up: If add-ons are installed through Paper UI, these are lost if the cache folder is deleted, see #319.

[kaikreuzer]

My nice theory from above does not really work out - by moving the etc folder from runtime to userdata, we now also have some system files in userdata that might need to be updated on an upgrade. Likewise, we might need to offer new versions of certain conf or userdata files that might have been edited by the user and in case we do folder structure refactorings, we might even have to move stuff around.

@theoweiss explained here how this is handled in Debian packages. Unfortunately, we require a mechanism that will work across all our packaging types (Debian, Docker, Synology, QNAP, Snap, etc.) and even on Windows 😦 .

Looking for good suggestions how to deal with that!

[mhilbush]

we now also have some system files in userdata that might need to be updated on an upgrade.

One thing I'll mention here, since I stumbled upon this just yesterday. I was not seeing in PaperUI any bindings that were newly added to the distro. It seems I got caught in the middle of this and a couple earlier changes.

#354

I discovered that userdata/config/profile.cfg was still referencing this:

repository.mvn:org.openhab.distro/openhab-features/2.0.0-SNAPSHOT/xml/features="mvn:org.openhab.distro/openhab-features/2.0.0-SNAPSHOT/xml/features"

instead of this:

repository.mvn:org.openhab.distro/openhab-aggregate-xml/2.0.0-SNAPSHOT/xml/features="mvn:org.openhab.distro/openhab-aggregate-xml/2.0.0-SNAPSHOT/xml/features"

I also noticed that the userdata/config directory (and contents) are not part of the distro. How is profile.config (and potentially other files in this directory) supposed to get updated when upgrading to a new distro?

[kaikreuzer]

userdata/config is the internal storage of OSGi ConfigurationAdmin, this must not be updated by anyone.
The "source" for these entries is the userdata/etc folder. You'll find the profile.cfg in there. This is one of the "system files" in userdata that I referred to above.

[BClark09]

Continuing conversation from openhab/openhab-docs#273

If all files that should be overwritten in /var/lib/openhab2/etc are listed somewhere in the distro, it should be possible for us to create upgrade .bat and .sh files for manual users, and also use this in the deb packaging.

Currently, I have made a batch script for windows users using a hard coded list (which I would appreciate people testing).

[xsnrg]

I thought about doing this with my shell script, but think it may be easier to just specify the ones that should not be touched as it is a much smaller list. This seems to be only the key based stuff, though I probably have not discovered everything. My current version of the script has a list in it like you are proposing, but I fear it is already out of date.

[kaikreuzer]

I wonder if there are any clever scripts/mechanisms out there, which would even allow us to define files that might have to be moved to a new folder or that might have to be deleted completely. Only being able to specify overwrite=yes/no might not be enough mid-term...

[xsnrg]

It is fairly easy to parse the directory tree from openhab/* down the tree looking for certain file names. If they are found in two locations, remove the older one. Is that the kind of thing you are thinking @kaikreuzer ? We would probably want to include etc/* in the name, provided the ones we are talking about are all in etc, of course. This may be easy in powershell as well, though I am not as versed there.

(edit) The delete part would be tricky without a list of files to be removed. Diff the folder structure against the distribution zip or tar note the deltas in certain places like the etc folder and remove non-matches. What could possibly go wrong? :)

Another option I have been pondering is the use of symlinks. Windows, Mac, and *nix systems all support them. This would allow us to put things that need to be kept, such as keys, in another location that cannot or does not get overwritten and then have the update script simply re-create the links for those files into the places openHAB looks for them at runtime.

[kaikreuzer]

Symlinks have a huge potential to mess up things - I would not use them for the "regular" installations.

Note that we might move files from userdata/etc to runtime/etc in future (if we manage to get Karaf accept it), so that we more clearly separate static app files from the user data. That's why I am asking for also a possibility to move files around, delete stuff etc.

[whopperg]

@kaikreuzer : i updated the script from the docs for me a little bit which you could see here:
openhab/openhab-docs#307 (comment)
this is just a starting point, but should get things rolling in the right direction...

[BClark09]

There's a couple of problems with that script in terms of sysVinit and systemd compatibility, but the rest seems good. See the debian scripts to see how it's done there. Essentially, you'd need to run the stop wait processes in prerm, continue on to the rest of your script and then run the start script found in postinst.

When we have an update.bat (which is done but needs testing) and an updated version of your update.sh in the runtime, we could then use an additional file to list, should be easy to extend from there.

[ThomDietrich]

Regarding the file list: I pretty much like the idea! This could be either done in a csv format or something more complex (json/xml). A csv (operation;file[;destination]) might already provide the needed flexibility and is easier parsable in low level languages like bash and batch. Any concerns?

[whopperg]

See the debian scripts to see how it's done there. Essentially, you'd need to run the stop wait processes in prerm, continue on to the rest of your script and then run the start script found in postinst.

Thanks! I will update the script - also we need to check which user is running the openhab service to set the correct permissions.

[whopperg]

Would it be possible if we dont have any version numbers in the urls or filenames at all? If so, this would be always consistent thru out the lifetime of openHAB2.

[kaikreuzer]

A csv (operation;file[;destination]) might already provide the needed flexibility

Sounds like a good suggestion (unless there is already some wide-spread format in use for such things?).

Would it be possible if we dont have any version numbers in the urls or filenames at all?

I'd think that we can avoid versions for most files (we won't be moving around bundle jars). If we have to add a version, we might also want to specifically target that version (i.e. delete file_1.1.txt, while keeping file_1.2.txt in place).

[ThomDietrich]

EDIT: Moved to PR #426

[bdleedy]

I wrote an upgrade script and posted it in documentation openhab/openhab-docs#365 only thinking that it would be a temporary help to users until the people that know what they're doing (you guys) could create the REAL solution. Well, it looks like mine might be a good start. I wrote it in Powershell and call it from a batch file for two reasons. One, you can right-click Run as admin and elevate the .ps1 (you can't right click on the .ps1 itself and do that), and two, we can temporarily bypass the Powershell execution policy. If needed, we could probably convert the entire thing back to batch but starting recently I've tried to write as much as I can in Powershell as Microsoft says they will be dropping the command prompt soon and many times it's more powerful and/or graceful.

I based it on what the documentation currently instructs here.

• Stop and delete the openHAB service.
• Prompt for the user to select the current install folder,
• Backup that folder
• Prompt for new .zip file
• Unzip to temp directory
• Delete files from current install that should not be kept (this could be converted to pull from.csv)
• Move new files into 'old' install
• Prompt if service is to be setup
• If so start the runtime, prompt the user to run the command, then call batch file.

I've noticed that there are other folders left behind that should be removed as well. for instance...\runtime\system\org\openhab\core\org.openhab.core now has 2.0.0 and 2.0.0-snapshot. These folders should be removed then as well. Is my script helpful?

[kaikreuzer]

Hi @whopperg, @bdleedy & all,

Sorry, I lost track of this, although it is pretty important.
As the openHAB 2.1 release is planned soon, I wonder if we can get some basic scripts working to get people from 2.0 release to 2.1 release easily. As it seems, we probably can do without all the csv magic for now, so your basic scripts for Linux & Windows should be a pretty good start.

Instead of following up on #426 and openhab/openhab-docs#365 individually, it is probably better to keep decisions here as a shared place.

For manually upgrading a 2.0 installation, I guess we have to offer the scripts as a separate download (as they aren't part of 2.0).

To keep it simple, I would simplify its logic and leave parts like shutting down the service and doing a backup to the user.

So we would mainly need these steps for a start:

• Download new release as zip to a temp folder
• Delete folders runtime, userdata/cache & userdata/tmp
• unzip the content of the release zip without overwriting any existing file

@whopperg I noticed that your update.sh script is very specific to Linux - it does not work on MacOS; i would be cool if we could have a script that works on both OSes, so that we do not need yet another version of the script.

What do you think? Can we get that working in the next days? Would be great!

[ThomDietrich]

I didn't know about Mac or RPM but there are more places where you'll find Upgrade, e.g. https://doc.owncloud.org/server/9.1/admin_manual/maintenance/upgrade.html

Anyhow, we can agree that it's more of a subjective topic and I support your decision. Let's leave it at update. 👍

[kaikreuzer]

I hope to have a pull request ready tomorrow.

@bdleedy Is "tomorrow" actually today or tomorrow (CET)? Would be great to have it by tonight CET, so that it can be tested and hopefully merged!

[bdleedy]

Hammering at this right now. Hoping in about an hour...

[BClark09]

Thanks @bdleedy, I'm online and monitoring this issue if you need any input from me.

[bdleedy]

Looks like it may be another 24 hours. Sorry, I'll keep you updated.

[BClark09]

Is there anything you can give us so far to help with or extend on, it would be great to include this onto the release?

[ghys]

I'm available to help with this as well if need be.

[kaikreuzer]

@bdleedy Sorry, not possible to wait - we will release today, so I need to do the final build this morning.

Could you share what you have so far so that @ghys could take over?

[martinvw]

@BClark09 I just did the upgrade but expected the update script to be in the openHAB directory after the update but it was not there is that expected?

Okay, even worse it reported success but broke my docker instance, the update does not seem to work on upgrading the docker instance to the next version, what I did was.

docker exec -it openhab2_openhab_1 /bin/bash

# installed curl

root@elbrus:/openhab# sh -c "$(curl -fsSL https://raw.githubusercontent.com/openhab/openhab-distro/master/distributions/openhab/src/main/resources/bin/update)" -- 2.1.0
                                         
#########################################
       openHAB 2.x.x update script       
#########################################
                                         
Making Temporary Directory
Downloading openHAB 2.1.0...
######################################################################## 100.0%
Update script in .zip archive found, using that instead.
                                         
#########################################
       openHAB 2.x.x update script       
#########################################
                                         
Making Temporary Directory
The script will attempt to update openHAB to version 2.1.0
Is this okay? [y/N]: y
Removing openHAB System Files...
Clearing cache...
Updating openHAB...
Deleting temporary files...
Restoring previous file ownership (openhab:openhab)

SUCCESS: openHAB updated from 2.1.0-SNAPSHOT to 2.1.0

root@elbrus:/openhab# ls
addons	conf  conf.dist  LICENSE.TXT  runtime  start.bat  start_debug.bat  start_debug.sh  start.sh  userdata  userdata.dist

[kaikreuzer]

Nobody ever claimed it would work for docker - it is only for "manual" installations!

[martinvw]

Nobody ever claimed it would work for docker - it is only for "manual" installations!

It is my 'production' environment and wanted to test the upgrade with my real sensors so I gave it a try

[BClark09]

Sorry, I don't know enough about docker to understand why this didn't work. It looks like it replaced the files correctly, but wouldn't know what those .dist directories are, anything out of the expected like this wouldn't work.

[martinvw]

Sorry, I don't know enough about docker to understand why this didn't work. It looks like it replaced the files correctly, but wouldn't know what those .dist directories are, anything out of the expected like this wouldn't work.

Nevermind most likely it is related to docker

[ghys]

@bdleedy about the PowerShell update script: due to time constraints - we needed something today - I quickly hacked together another one by taking some bits of @BClark09's original script and following (part of) the logic in the shell script.

The result was merged earlier today in #473.

Just know my intention wasn't at all to thwart your effort; in fact, there are a few missing pieces in this script, so if what you have already worked on has more features or do things better, I would glady collaborate with you to come up with a final, complete script. 🤝

[bdleedy]

@ghys No worries. It looks very similar to my script in many ways. I have a few subtle differences and some larger ones. I was trying to use ExtractToFile but couldn't get that to create folders while not overwriting. I was going to go a route more like yours. I don't see how yours doesn't overwrite though. I also kept as many of my variables and Write-Host outputs the same as the other script for ease of editing later. I'll try to put the two together and drop a message with it in the PR.

[zarusz]

I am trying to migrate from OpenHAB 2.0.0-1 to 2.1 on an raspbian according to this instruction. Invoking the upgrade command:

sudo sh -c "$(curl -fsSL https://raw.githubusercontent.com/openhab/openhab-distro/master/distributions/openhab/src/main/resources/bin/update)" -- 2.1.0

Yelds the following error:

The script must be called from openHAB's root directory.
Either specify a directory or place this update script in and run from openHAB's root folder.

Looking at the script, it does not seem to account for my particular installation layout, which is:
/etc/openhab2
/var/lib/openhab2
/usr/share/openhab2

The script was executed from within /usr/share/openhab2 which I guess is the root in that case.

Am I doing something wrong or is the script not meant for 2.0 ? Any thoughts?

[whopperg]

@zarusz
Hi,

Sure that you got the manual installation of openHAB?

Looks like you installed it via apt-get or a deb file, I which case this doesn't work.

If you want to use this, the Installation has to be manual( ie: there is only one path under /opt/openhab2)!

Cheers

[BClark09]

Hi @zarusz, the script is for any manual version. It looks like you installed openHAB using apt-get.

If you're still on 2.0 you should simply be able to update it as if it was any other software on Raspbian using

apt-get update && apt-get upgrade

If this does not work, then you probably have removed the repository information at some point, just enter in these commands:

wget -qO - 'https://bintray.com/user/downloadSubjectPublicKey?username=openhab' | sudo apt-key add -
sudo apt-get install apt-transport-https
echo 'deb https://dl.bintray.com/openhab/apt-repo2 stable main' | sudo tee /etc/apt/sources.list.d/openhab2.list
apt-get update && apt-get upgrade

[whopperg]

@BClark09 and @ALL,

Thanks for taking over! I'm just back from Holidays.
Looks really good to me!

Do I have to do something or are we good here?

[zarusz]

Hi @BClark09 and @whopperg,

Thanks for you prompt response ;)

Yes, indeed I do recall installing it from apt-get in the first place (sorry my bad). Yet, somehow apt-get update && apt-get upgrade did not work for me so I figured maybe the script will help. However, I just followed @BClark09 script and it works now!

Also I just had to add sudo in the last line for it to work:

wget -qO - 'https://bintray.com/user/downloadSubjectPublicKey?username=openhab' | sudo apt-key add -
sudo apt-get install apt-transport-https
echo 'deb https://dl.bintray.com/openhab/apt-repo2 stable main' | sudo tee /etc/apt/sources.list.d/openhab2.list
sudo apt-get update && sudo apt-get upgrade

Thanks again!

[BClark09]

Glad you got it working @zarusz!

@whopperg, hope you had a good vacation! There's always something to do around here! ;) In particular I think it'd be good to bundle the next versions of openHAB with a backup and restore scripts. Anyone from this thread should correct me if I'm wrong, but I think the list of things to do for this issue at the moment are as follows:

• Create a backup script (you've pretty much done most of this already!).
  • This should be able to work regardless of installation type (even apt-get!).
• Create a restore script.
  • This should be able to work regardless of installation type (even apt-get!).
• Update the windows update script to the same level of functionality as the Linux script.
  • Allow update of addons files.
  • Possibly make the script compatible with all versions of powershell or even cmd prompt.
• Finalise the idea a csv system to prepare for the update of Karaf.
  • The config files should be moved from A to B, and the linuxpkgs should also use it.

[BClark09]

Hi all, will it make sense to make a new issue for each of the 4 points above? This thread is rather long and may actions have been taken.

[kaikreuzer]

Good idea!

[kaikreuzer]

Further discussion on the three issues linked above.