Update recommendations and troubleshooting

If something goes wrong with the update, it is in almost all cases related to the Apache or Nginx configuration on the host, and NOT Docker and the running containers!

If you are not familiar with your Apache/Nginx configuration (including https), or need to ensure only a minimal downtime, the recommendation will be always to do the painless new installation on a new host and then an migration of your database and files to the new host! Same is true if your old hosts Linux version is no longer supported from it's distribution.

Requirements for the update to succeed

The update operates under the assumption you have NOT changed any of the following:

files or backup directory (/var/lib/egroupware/default/files or /var/lib/egrouware/default/backup)
your database runs on the host AND you use "localhost" as "db_host in header.inc.php
your EGroupware runs under http(s)://your.domain/egroupware (Setup >> Konfiguration >> Webserver URL: /egroupware).
with the exception of MariaDB/MySQL no other service (LDAP, IMAP, PostgreSQL) uses a host-name of localhost

If you changed anything of the above, you either need to change it back to the default locations, or you need to fix your installation after the update. EGroupware will fail after the update without any further changes!

How to migrate your data from an existing 17.1 or older installation

do a new installation as described for the distribution of the new host
enable https using Let's Encrypt following the exzellent instructs from eg. Digital Ocean for Apache or Nginx or take a look at SSL Encryption with Let’s Encrypt in the forum
make a database backup inside your old EGroupware using Administration >> Database backup and restore
rsync the files and backups from the old installation to the new server, run on the new server the following commands:

rsync -av --delete root@<ip-of-old-server>:/var/lib/egroupware/default/ /var/lib/egroupware/default/
chown -R 33:33 /var/lib/egroupware/default

log into the new EGroupware using the generated sysop user and go to Administration >> Database backup and restore and restore the last backup you rsynced to the new server
you can repeat steps 3. to 5. multiple times, until you are ready to use the new system productively

Changing the network address-range of Docker

Docker and docker-compose creates (internal) networks by default in the 172.[17-31].0.0 range. If you use this range already for existing networks you need to change the default BEFORE installing EGroupware, as you wont be able to reach EGroupware from 172.17+ networks!

To move all Docker IP addresses / networks eg. to 172.22.0.0/16 you need to do the following: Create the file /etc/docker/daemon.json with the following content:

{
  "default-address-pools":
  [
    {"base":"172.22.0.0/16","size":24}
  ]
}

Then you need to restart the Docker daemon, if you already installed it: systemctl restart docker

In case you already installed EGroupware you need to delete all containers and networks, as the above only works for newly created networks NOT for the existing ones:

for dir in /etc/egroupware-*
do
    cd $dir
    docker-compose stop
    docker-compose rm -f
done
docker system prune
for dir in /etc/egroupware-*
do
    cd $dir
    docker-compose up -d
done

If your Linux server needs to use a proxy to access the internet

a) company proxy on an other host or

b) a proxy tunneled in via ssh: ssh -R 9090:localhost:9090 egw.example.org

Following config assumes the proxy is running under localhost:9090, replace that with your host, port and - if necessary credentials (user:pass@host:port)

Configured your Linux system so it can install and update packages via your proxy:

# /etc/apt/apt.conf.d/proxy.conf
Acquire::http::Proxy "http://localhost:9090";

set environment variables eg. in ~/.bashrc to get eg. curl working:

export http_proxy=http://localhost:9090
export https_proxy=http://localhost:9090
# to import in your running shell use: . ~/.bashrc

Docker needs to have the proxy configured to be able to pull images:

# /etc/systemd/system/docker.service.d/http-proxy.conf
[Service]
Environment="HTTP_PROXY=http://localhost:9090"
Environment="HTTPS_PROXY=http://localhost:9090"
#Environment="NO_PROXY=localhost,127.0.0.1,docker-registry.example.com,.corp"
# the following will refresh your running docker daemon: systemctl daemon-reload && systemctl restart docker

EGroupware can also use a proxy to eg. check for an update. Proxy settings are in Setup (https://example.com/egroupware/setup/) upper login under: Edit current configuration > Host information

Trouble-shooting a not working installation after the update

What to do if you can not access EGroupware after the update:

run docker ps to check all containers are running:

root@debian:~# docker ps
CONTAINER ID        IMAGE                                         COMMAND                  CREATED             STATUS                            PORTS                      NAMES
a6cb25ad6b2d        nginx:stable-alpine                           "/docker-entrypoint.…"   29 minutes ago      Up 29 minutes                     127.0.0.1:8080->80/tcp     egroupware-nginx
290fd2abed44        phpswoole/swoole:latest                       "/entrypoint.sh"         29 minutes ago      Up 29 minutes                                                egroupware-push
d8febd271620        download.egroupware.org/egroupware/epl:20.1   "/entrypoint.sh php-…"   29 minutes ago      Up 29 minutes                     9000/tcp                   egroupware
1a6a24e51144        containrrr/watchtower                         "/watchtower --sched…"   5 weeks ago         Up 2 hours                        8080/tcp                   egroupware-watchtower
bbc8657ad981        rocketchat/rocket.chat:latest                 "docker-entrypoint.s…"   2 hours ago         Up About a minute                 127.0.0.1:3000->3000/tcp   rocketchat
1bc8a8f0eae8        mongo:4.0                                     "docker-entrypoint.s…"   2 hours ago         Restarting (100) 38 seconds ago                              rocketchat-mongo
0a1d2b2257c8        quay.io/egroupware/collabora-key:latest       "/bin/sh -c 'bash st…"   2 hours ago         Up 2 hours                        127.0.0.1:9980->9980/tcp   collabora-key

(If you only installed EGroupware and not Collabora and Rocket.Chat, you should have only the first 4 containers with names starting with egroupware.

Look at the logs:

cd /etc/egroupware-docker
docker-compose logs -f
root@debian:/etc/egroupware-docker# docker-compose logs
Attaching to egroupware-nginx, egroupware, egroupware-watchtower
egroupware    | Fix APC(u) configuration, set apc.shm_size=128M in /etc/php/7.3/cli/conf.d/20-apcu.ini
egroupware    |
egroupware    | EGroupware successful updated
egroupware    |  * Starting periodic command scheduler cron
egroupware    |    ...done.
egroupware    |
egroupware    | EGroupware successful updated
egroupware    |
egroupware    | EGroupware successful updated
egroupware    |
egroupware    | EGroupware successful updated
egroupware    | [06-Sep-2019 10:41:13] NOTICE: fpm is running, pid 1
egroupware    | [06-Sep-2019 10:41:13] NOTICE: ready to handle connections
egroupware    | [06-Sep-2019 10:41:13] NOTICE: systemd monitor interval set to 10000ms
egroupware-watchtower | time="2019-09-15T07:11:12Z" level=info msg="Waiting for running update to be finished..."

If you don't see the line fpm is running, pid 1 you will will only get a 502 Bad Gateway from Nginx, when you try to connect to EGroupware. The startup of the egroupware container might take a little longer then you expect, as it has to copy the EGroupware sources into a volume which is shared with Nginx, so the webserver can deliver the static content (CSS, JavaScript, images, ...).

If egroupware container alone was restarted without Nginx, Nginx still uses the old IP of egroupware and you get a 502 Bad Gateway. Always restart both: (cd /etc/egroupware-docker; docker-compose restart)

Try to connect to EGroupware container without using the proxy running on the host webserver. We use a tool called curl for these kind of checks. Install it with apt install -y curl (or yum or zypper instead of apt depending on your distribution):

root@debian:/etc/egroupware-docker# curl -i http://localhost:8080/egroupware/
HTTP/1.1 302 Found
Server: nginx/1.16.1
Date: Sun, 15 Sep 2019 07:19:10 GMT
Content-Type: text/html; charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive
Cache-Control: no-store, no-cache, must-revalidate
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Pragma: no-cache
Location: /egroupware/login.php?phpgw_forward=%252Findex.php

If you see the above EGroupware containers are running, it is installed and has a working database connection.

Try accessing EGroupware through the host webserver / proxy possibly with https, if you configured that already:

RalfsMac:epl-19.1 ralf$ curl -i https://testbox9.egroupware.org/egroupware/
HTTP/1.1 302 Found
Date: Sun, 15 Sep 2019 07:26:42 GMT
Server: nginx/1.16.1
Content-Type: text/html; charset=UTF-8
Cache-Control: no-store, no-cache, must-revalidate
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Pragma: no-cache
Location: /egroupware/login.php?phpgw_forward=%252Findex.php
Transfer-Encoding: chunked

If you can see the correct output of 3., but not the above one, something is wrong with your proxy configuration on the host.

Check the push server and fix problems with it

Push server container (egroupware-push) is new in 20.1 and hold a websocket connections to each connected browser.

It allows the EGroupware server-side to push updates to every user online.

One way to test it's working is to login with the same account into EGroupware from two different computers, browsers or using an anonymous tab. Go to InfoLog on both and add a new InfoLog on one. It should instantly appear on the other one too.

Not all apps implement push (with 20.1 release it's Calendar, InfoLog and Timesheet, with 21.1 Tracker and Addressbook too) or need special configuration like Mail app on the used IMAP server.

If it does not, open developer tools in your browser, go to the networking tab, reload (F5), search for push and click on Messages: The green up arrow you see shows the websocket connection from your browser to the push server. A red down arrow shows messages pushed to the browser by EGroupware server-side, eg. when you add an InfoLog entry in the above test.

If it does NOT show messages or type websocket with status 101, you have a configuration problem on your server, probably on the webserver used as reverse proxy on the host itself (NOT in a container!):

first check with docker ps that you have a running egroupware-push container
use curl -i https://example.org/egroupware/push (replacing example.org with your domain!) it should output something like:

HTTP/1.1 401 Unauthorized
Server: nginx/1.18.0
Date: Wed, 19 Aug 2020 16:05:45 GMT
Content-Type: text/html
Content-Length: 21
Connection: keep-alive
Www-Authenticate: Basic realm="EGroupware Push Server"

Missing Bearer Token!

if you get 404 Not Found you have not the latest webserver configuration (/etc/egrouwpare-docker/nginx.conf or apache2.conf), eg. because you made some changes and the package update did not overwrite them. In that case look for a .dpkg-dist or .rpmnew file with the same name, add your changes to that file and rename it to just nginx.conf or apache2.conf. Then you need to reload your webserver with nginx -s reload or systemctl restart apache2 (or httpd on RHEL/CentOS).
if curl complains about the certificate, you either need to use an officially accepted one (eg. Let's Encrypt) or you need to make the CA used to sign the certificate available to the containers, which need to talk to each other (EGroupware, Collabora, Rocket.Chat, Guacamole, ...). The later is a lot of work and not recommended, since Let's Encrypt is available. There are commented out commands in each docker-compose.override.yml to make the CA known to the containers.
if you get the above output, but the not the connection in the browser (green array with {"subscribe":...

if you use Apache make sure /etc/egroupware-docker/apache2.conf is included in the used ssl VirtualHost, including it outside the vHost does NOT work for websockets in Apache!
if you use eg. your firewall to terminate https or an other proxy, you have to enable websocket support!

if you see the green array, but no red one(s) when you eg. add an InfoLog, run the above curl command inside the egroupware container to see if you get the same output as above: docker exec -it egroupware curl -i https://example.org/egroupware/push

curl: (6) Could not resolve host: example.org or curl: (7) Failed to connect to example.org port 443: Connection refused either your (externally) used host-name could not be resolved in the container or your firewall does not allow access from the inside to your external address. Use an extra_hosts entry in your /etc/egroupware/docker-compose.override with your docker0 address to which your webserver on the host is also listening:

service:
  egroupware:
    extra_hosts:
    - "example.org:172.17.0.1"

curl: (60) SSL: no alternative certificate subject name matches target host name 'example.org' you use a self-signed certificate and need to install your CA via /etc/egroupware-docker/docker-compose.override.yml in the egroupware container
Collabora and Rocket.Chat also need to be able to access the EGroupware container and therefore need a similar fix see their docker-compose.yml files!

check the log of the push server: docker logs -f egroupware-push

if you only see server: handshake success with fd****, FAILED with unknown sessionid=********** check that egroupware and egroupware-push container share the session volume:

docker exec -it egroupware ls /var/lib/php/sessions
docker exec -it egroupware-push ls /var/lib/php/sessions

both should show exactly the same session files, as they mount the same value

EGroupware container can not access MariaDB/MySQL

This should not happen any more with 20.1, unless you updated from 19.1, as 20.1 runs MariaDB 10.4 in a container for new installations.

egroupware    | /usr/bin/php7.3 -d memory_limit=-1 /usr/share/egroupware/setup/setup-cli.php --update 'all,admin,HU4M0zQixh5:fda!'
egroupware    | EGroupware API version 19.1 found.
egroupware    | EGroupware configuration file (header.inc.php) version 1.29 exists and is up to date
egroupware    | Your database is not working! mysqli://egroupware:*********@localhost/egroupware:
egroupware    |
egroupware    | Installation failed --> exiting!
egroupware    |
egroupware    | Retrying EGroupware installation in 3 seconds ...

That situation can happen in a couple of different ways:

RHEL/CentOS does not start or enable the DB by default, you have to do so explicitly!
DB was shut down manually or also updated at the same time as EGroupware

The MariaDB/MySQL socket is bind-mounted into the egroupware container so db_host="localhost" continues to work.

If the DB is not running and therefore the socket is not there when EGroupware container starts, Docker creates a directory with the same name on the host AND inside the container.

Neither of them does recover automatically from that situation!

You have to manually fix it:

cd /etc/egroupware-docker
docker-compose stop egroupware
docker-compose rm -f egroupware
rm -rf /var/{run,lib}/mysql*/mysql*.sock
systemctl start mariadb # or mysqld
systemctl enable mariadb # only necessary for RHEL/CentOS not doing so by installing the DB
systemctl status mariadb
docker-compose up -d

Docker on Btrfs

Running Docker on a Btrfs partition causes a problem: Old images are not deleted and the partition fills up over time. Here is a (long) thread on this topic:

https://github.com/moby/moby/issues/27653

So it is currently (12.2019) not recommended to run Docker on Btrfs. Otherwise, regular checks and manual cleanup are necessary.

If someone has a solution, we ask for feedback in the forum.

PHP FPM status page - see in realtime the requests on your EGroupware server

The PHP FPM status page shows, if enabled, the currently running PHP requests incl. their URL.

This helps to troubleshoot a slow or overloaded server.

Instructions on how to enable it are in the file itself:

https://github.com/EGroupware/egroupware/blob/master/admin/fpm-status.php

The status-page will then be available for logged in EGroupware administrator under your EGroupware URL:

https://example.org/egroupware/admin/fpm-status.php

Before version 23.1 the status page needs to be manually copied into your EGroupware container:

curl https://github.com/EGroupware/egroupware/raw/master/admin/fpm-status.php > /tmp/fpm-status.php
docker cp /tmp/fpm-status.php egroupware:/usr/share/egroupware-sources/admin/
docker restart egroupware