Upgrade PostgreSQL to v16

[enoch85]

Just tested, and seems to work!

[netlify]

✅ Deploy Preview for teslamate ready!

Name	Link
🔨 Latest commit	0dcabf6
🔍 Latest deploy log	https://app.netlify.com/sites/teslamate/deploys/665106c13f789b00085ed6bb
😎 Deploy Preview	https://deploy-preview-3884--teslamate.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[Dulanic]

While I agree changing from v15 to v16 makes sense, please be aware this has breaking changes and simply upgrading in many cases will cause breaks and you will need to do a pg_dumpall or use pg_uprade so we need to be aware if a user just upgrades their container, it may cause issues.

https://www.postgresql.org/docs/16/release-16.html#RELEASE-16-MIGRATION

[enoch85]

pg_uprade

Yeah that could be added to the documented steps of upgrading PostgreSQL as well. Want me to fix? Not 100% sure about the command, but will check.

JFTR, I checked all my statistics from 2022 --> now, and they seem to work, everything is there afaik. Will do a pg_upgradenow just in case though, good advice! 👍

[enoch85]

OK, so it wasn't as easy as I thought. Will have to revert now, and do it properly with pg_dumpall instead.

Maybe the standard command recomended in documentation should be pg_dumpall instead of just pg_dump?

[enoch85]

I got this in the log now which I didn't before, don't know if it makes any sense? With pg_dumpI got no errors, but then maybe you don't need to recreate the tables either, since you're doing a dumpall? https://docs.teslamate.org/docs/maintenance/backup_restore#restore

2024-05-06 15:29:59.173 CEST [1] LOG:  starting PostgreSQL 16.2 (Debian 16.2-1.pgdg120+2) on x86_64-pc-linux-gnu, 
.......................
2024-05-06 22:38:33.499 CEST [86] ERROR:  role "teslamate_db_user" already exists
2024-05-06 22:38:33.499 CEST [86] STATEMENT:  CREATE ROLE teslamate_db_user;
2024-05-06 22:38:33.563 CEST [88] ERROR:  database "teslamate" already exists
2024-05-06 22:38:33.563 CEST [88] STATEMENT:  CREATE DATABASE teslamate WITH TEMPLATE = template0 ENCODING = 'UTF8' LOCALE = 'en_US.utf8';
``

[enoch85]

All good from me now anyway! 👍

[Dulanic]

Now you understand the pain 🤣 I personally run a separate postgres server as I have many containers connect so I just transition when I feel like it. pg_upgrade doesn't work too well with container postgres.

[enoch85]

At least this PR could now be merged as it does what proposed - switch to PGSQL 16 and upgrade properly.

[parkr]

What are the benefits of 16 over 15? As a user with several years of data, I'd be hesitant to upgrade unless there were benefits.

[enoch85]

What are the benefits of 16 over 15? As a user with several years of data, I'd be hesitant to upgrade unless there were benefits.

Well, newer = better. :) I haven't gone into details, but updating is the best protection against old and unsecure code.

Since it's "easy" to upgrade, why not? I also think that the default for new installs should be the latest version.

[enoch85]

@JakobLichterfeld What do you say, can I get your approval on this?

[JakobLichterfeld]

Thanks for your suggestion!
I knew, the time will come to update our db container. I'm with Dulanic, this need additional testing (our CI does not cover backup and restore for example), given the fact some users are still on very old Postgres versions (as we can see in the amount users who post their docker-compose.yml in issues).

[enoch85]

Thanks for your suggestion! I knew, the time will come to update our db container. I'm with Dulanic, this need additional testing (our CI does not cover backup and restore for example), given the fact some users are still on very old Postgres versions (as we can see in the amount users who post their docker-compose.yml in issues).

Hmm, I don't see the issue here actually. You set a tag in your docker-compose.yaml, and as long as that's set, nothing will change - the version will stay the same.

If you on the other hand put a specific version (in this case 16) in the docker-compose.yaml file, it will install with 16, and stay as 16 until you change it.

This PR merly suggests that all new users should get 16 directly, instead of 15 as it is today. It won't update current users.

I run PostgreSQL with your Docker setup, I don't have any external DB, and I managed to upgrade according to your documented instructions. Personally, I went from 14 to 16 in one go. 🙂

[JakobLichterfeld]

Obviously, you get me wrong. Current state is: some users didn't upgrade Postgres every, they stick with old Postgres version. When they post their docker-compose, we often redirect them to our upgrade guide to be future-proof on that end, even if unrelated to the issue they report.

If we change official installation instructions, we must ensure the manual upgrade process runs smooth, in-depended on the version they come from, this can be Postgres 11 for example.

pg_dump and pg_dumpall are not the same. May you elaborate why changing makes sense? What are the consequences, what are the differences in our migration?

I would love to upgrade to Postgres 16, but we need to ensure all is working as expected, as, as said above, our CI does not cover these cases.

[brianmay]

I am guessing that pg_dumpall will save some details (e.g. database user) that may not be recorded by pg_dump, but you probably do want to keep. Not tested though.

[enoch85]

From the docs (https://www.postgresql.org/docs/current/app-pgdump.html):

pg_dump only dumps a single database. To back up an entire cluster, or to back up global objects that are common to all databases in a cluster (such as roles and tablespaces), use pg_dumpall.

As I wrote above, I managed to upgrade to v16 with pg_dump but then @Dulanic made me worried I missed somthing by not running pg_dumpall - so I ended up with pg_dumpall even though both seemed to be working totally fine (I checked history from day one, everything was there afaics). My educated guess though is that pg_dump would be sufficient since you advice to create new tables and everything before importing. pg_dumpall will backup the whole PostgreSQL DB (all of them), including users and everything - and that's not needed since we are only looking for the data in one single database - teslamate. But, better safe than sorry, hence the commit.

[enoch85]

(Migrated) PostgreSQL 16 is still running fine. ;)

[swiffer]

adrian did the same last year (bump to 15).

b79a454

i think it's fine to bump for new users.

[BowlesCR]

Fwiw, I've been running on Postgres 16-alpine since mid-February with no issues beyond the requisite backup/restore to migrate.

At risk of starting a tangential discussion, debates to be had on using the alpine variant vs the default bookworm (actually bookworm-slim in the Dockerfile)... Without digging into the numbers the size reduction of alpine (only ~200MB) might be negated by no longer sharing the bookworm-slim base layer between the app and DB. That said, our Grafana image is based on alpine as well, so 🤷

[parkr]

Seems useful to have new users start on the latest version. I'd recommend splitting this PR into (1) the version change and (2) the backup/restore documentation change.

FWIW I did the upgrade to 16 (from 13) following the existing instructions (no dumpall) and that worked well for me.

[enoch85]

Seems useful to have new users start on the latest version. I'd recommend splitting this PR into (1) the version change and (2) the backup/restore documentation change.

FWIW I did the upgrade to 16 (from 13) following the existing instructions (no dumpall) and that worked well for me.

Yeah, worked flawlessly for me as well, so I don't see any reason to use dumpall really, but I got insecure as mentioned in previous posts.

[JakobLichterfeld]

In addition to my review findings: Please also adapt all other mentions in the docu.

[enoch85]

In addition to my review findings: Please also adapt all other mentions in the docu.

Done! 👍

[JakobLichterfeld]

ty!

[JakobLichterfeld]

CI is now updated as well, see #3916

[enoch85]

Great! 🎊