Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

In certain cicumstances, created certificates are deleted on next round #5834

Open
5 tasks done
Korsani opened this issue Apr 8, 2024 · 4 comments
Open
5 tasks done

In certain cicumstances, created certificates are deleted on next round #5834

Korsani opened this issue Apr 8, 2024 · 4 comments
Labels
bug

Comments

@Korsani
Copy link

Korsani commented Apr 8, 2024

Contribution guidelines

I've found a bug and checked that ...

  • ... I understand that not following the below instructions will result in immediate closure and/or deletion of my issue.
  • ... I have understood that this bug report is dedicated for bugs, and not for support-related inquiries.
  • ... I have understood that answers are voluntary and community-driven, and not commercial support.
  • ... I have verified that my issue has not been already answered in the past. I also checked previous issues.

Description

In mailcow/acme under certain circumstances created certificates are deleted on the next certificate update

Logs:

no logs

Steps to reproduce:

1 - Try /srv/acme.sh with forcing "bl-evolution.com" in SQL_DOMAINS (add `SQL_DOMAINS="bl-evolution.com` between line 227 and 228)
2 - Constate that there is `autoconfig.bl-evolution.com` in ssl directory (`/opt/mailcow-dockerized/data/assets/ssl`)
3 - Rerun /src/acme.sh : `autoconfig.bl-evolution.com` should have disappeared

Which branch are you using?

master

Which architecture are you using?

x86

Operating System:

Debian Bookworm

Server/VM specifications:

32G ram, Intel(R) Xeon(R) CPU D-1520 @ 2.20GHz

Is Apparmor, SELinux or similar active?

don't know

Virtualization technology:

don't know

Docker version:

24.0.6

docker-compose version or docker compose version:

v2.21.0

mailcow version:

2024-02

Reverse proxy:

nginx

Logs of git diff:

none

Logs of iptables -L -vn:

# Warning: iptables-legacy tables present, use iptables-legacy to see them
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
  37M  135G MAILCOW    0    --  *      *       0.0.0.0/0            0.0.0.0/0            /* mailcow */
 790M 2826G DOCKER-USER  0    --  *      *       0.0.0.0/0            0.0.0.0/0
 790M 2826G DOCKER-ISOLATION-STAGE-1  0    --  *      *       0.0.0.0/0            0.0.0.0/0
 670M 2597G ACCEPT     0    --  *      br-mailcow  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
9581K  653M DOCKER     0    --  *      br-mailcow  0.0.0.0/0            0.0.0.0/0
 111M  228G ACCEPT     0    --  br-mailcow !br-mailcow  0.0.0.0/0            0.0.0.0/0
9435K  645M ACCEPT     0    --  br-mailcow br-mailcow  0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     0    --  *      docker0  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
    0     0 DOCKER     0    --  *      docker0  0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     0    --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     0    --  docker0 docker0  0.0.0.0/0            0.0.0.0/0

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 ACCEPT     6    --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.5           tcp dpt:8983
    0     0 ACCEPT     6    --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.6           tcp dpt:3306
    0     0 ACCEPT     6    --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.8           tcp dpt:8443
    0     0 ACCEPT     6    --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.8           tcp dpt:8080
    0     0 ACCEPT     6    --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.250         tcp dpt:12345
    7   360 ACCEPT     6    --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.250         tcp dpt:4190
 1983  119K ACCEPT     6    --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.250         tcp dpt:995
 3369  194K ACCEPT     6    --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.250         tcp dpt:993
  368 21868 ACCEPT     6    --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.250         tcp dpt:143
  366 21776 ACCEPT     6    --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.250         tcp dpt:110
  933 55132 ACCEPT     6    --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.253         tcp dpt:587
 2062  123K ACCEPT     6    --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.253         tcp dpt:465
13017  776K ACCEPT     6    --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.253         tcp dpt:25
    0     0 ACCEPT     6    --  !br-mailcow br-mailcow  0.0.0.0/0            172.22.1.249         tcp dpt:6379

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
 pkts bytes target     prot opt in     out     source               destination
 111M  228G DOCKER-ISOLATION-STAGE-2  0    --  br-mailcow !br-mailcow  0.0.0.0/0            0.0.0.0/0
    0     0 DOCKER-ISOLATION-STAGE-2  0    --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0
1346M 3184G RETURN     0    --  *      *       0.0.0.0/0            0.0.0.0/0

Chain DOCKER-ISOLATION-STAGE-2 (2 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DROP       0    --  *      br-mailcow  0.0.0.0/0            0.0.0.0/0
    0     0 DROP       0    --  *      docker0  0.0.0.0/0            0.0.0.0/0
 214M  269G RETURN     0    --  *      *       0.0.0.0/0            0.0.0.0/0

Chain DOCKER-USER (1 references)
 pkts bytes target     prot opt in     out     source               destination
1346M 3184G RETURN     0    --  *      *       0.0.0.0/0            0.0.0.0/0

Chain MAILCOW (1 references)
 pkts bytes target     prot opt in     out     source               destination
  803 48100 DROP       0    --  *      *       194.169.175.10       0.0.0.0/0
    0     0 DROP       6    --  !br-mailcow br-mailcow  0.0.0.0/0            0.0.0.0/0            /* mailcow isolation */

Logs of ip6tables -L -vn:

# Warning: ip6tables-legacy tables present, use ip6tables-legacy to see them
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
4201K 3802M MAILCOW    0    --  *      *       ::/0                 ::/0                 /* mailcow */
  88M  103G DOCKER-USER  0    --  *      *       ::/0                 ::/0
  88M  103G DOCKER-ISOLATION-STAGE-1  0    --  *      *       ::/0                 ::/0
  56M   54G ACCEPT     0    --  *      br-mailcow  ::/0                 ::/0                 ctstate RELATED,ESTABLISHED
2563K  191M DOCKER     0    --  *      br-mailcow  ::/0                 ::/0
  30M   48G ACCEPT     0    --  br-mailcow !br-mailcow  ::/0                 ::/0
2353K  174M ACCEPT     0    --  br-mailcow br-mailcow  ::/0                 ::/0
    0     0 ACCEPT     0    --  *      docker0  ::/0                 ::/0                 ctstate RELATED,ESTABLISHED
    0     0 DOCKER     0    --  *      docker0  ::/0                 ::/0
    0     0 ACCEPT     0    --  docker0 !docker0  ::/0                 ::/0
    0     0 ACCEPT     0    --  docker0 docker0  ::/0                 ::/0

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination
    1    64 ACCEPT     6    --  !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::b  tcp dpt:4190
   87  6944 ACCEPT     6    --  !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::b  tcp dpt:995
 5911  475K ACCEPT     6    --  !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::b  tcp dpt:993
  187 15788 ACCEPT     6    --  !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::b  tcp dpt:143
   21  1512 ACCEPT     6    --  !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::b  tcp dpt:110
    7   544 ACCEPT     6    --  !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::10  tcp dpt:587
  255 18776 ACCEPT     6    --  !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::10  tcp dpt:465
  453 35196 ACCEPT     6    --  !br-mailcow br-mailcow  ::/0                 fd4d:6169:6c63:6f77::10  tcp dpt:25

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
 pkts bytes target     prot opt in     out     source               destination
  30M   48G DOCKER-ISOLATION-STAGE-2  0    --  br-mailcow !br-mailcow  ::/0                 ::/0
    0     0 DOCKER-ISOLATION-STAGE-2  0    --  docker0 !docker0  ::/0                 ::/0
 213M  296G RETURN     0    --  *      *       ::/0                 ::/0

Chain DOCKER-ISOLATION-STAGE-2 (2 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DROP       0    --  *      br-mailcow  ::/0                 ::/0
    0     0 DROP       0    --  *      docker0  ::/0                 ::/0
  57M   76G RETURN     0    --  *      *       ::/0                 ::/0

Chain DOCKER-USER (1 references)
 pkts bytes target     prot opt in     out     source               destination
 213M  296G RETURN     0    --  *      *       ::/0                 ::/0

Chain MAILCOW (1 references)
 pkts bytes target     prot opt in     out     source               destination

Logs of iptables -L -vn -t nat:

# Warning: iptables-legacy tables present, use iptables-legacy to see them
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
8853K  410M DOCKER     0    --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
65546 2884K DOCKER     0    --  *      *       0.0.0.0/0           !127.0.0.0/8          ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
6051K  500M MASQUERADE  0    --  *      !br-mailcow  172.22.1.0/24        0.0.0.0/0
    3   180 MASQUERADE  0    --  *      !docker0  172.17.0.0/16        0.0.0.0/0
    0     0 MASQUERADE  6    --  *      *       172.22.1.5           172.22.1.5           tcp dpt:8983
    0     0 MASQUERADE  6    --  *      *       172.22.1.6           172.22.1.6           tcp dpt:3306
    0     0 MASQUERADE  6    --  *      *       172.22.1.8           172.22.1.8           tcp dpt:8443
    0     0 MASQUERADE  6    --  *      *       172.22.1.8           172.22.1.8           tcp dpt:8080
    0     0 MASQUERADE  6    --  *      *       172.22.1.250         172.22.1.250         tcp dpt:12345
    0     0 MASQUERADE  6    --  *      *       172.22.1.250         172.22.1.250         tcp dpt:4190
    0     0 MASQUERADE  6    --  *      *       172.22.1.250         172.22.1.250         tcp dpt:995
    0     0 MASQUERADE  6    --  *      *       172.22.1.250         172.22.1.250         tcp dpt:993
    0     0 MASQUERADE  6    --  *      *       172.22.1.250         172.22.1.250         tcp dpt:143
    0     0 MASQUERADE  6    --  *      *       172.22.1.250         172.22.1.250         tcp dpt:110
    0     0 MASQUERADE  6    --  *      *       172.22.1.253         172.22.1.253         tcp dpt:587
    0     0 MASQUERADE  6    --  *      *       172.22.1.253         172.22.1.253         tcp dpt:465
    0     0 MASQUERADE  6    --  *      *       172.22.1.253         172.22.1.253         tcp dpt:25
    0     0 MASQUERADE  6    --  *      *       172.22.1.249         172.22.1.249         tcp dpt:6379

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination
   31  1860 RETURN     0    --  br-mailcow *       0.0.0.0/0            0.0.0.0/0
    0     0 RETURN     0    --  docker0 *       0.0.0.0/0            0.0.0.0/0
    0     0 DNAT       6    --  !br-mailcow *       0.0.0.0/0            127.0.0.1            tcp dpt:18983 to:172.22.1.5:8983
    0     0 DNAT       6    --  !br-mailcow *       0.0.0.0/0            127.0.0.1            tcp dpt:13306 to:172.22.1.6:3306
    0     0 DNAT       6    --  !br-mailcow *       0.0.0.0/0            127.0.0.1            tcp dpt:8443 to:172.22.1.8:8443
    0     0 DNAT       6    --  !br-mailcow *       0.0.0.0/0            127.0.0.1            tcp dpt:8080 to:172.22.1.8:8080
    0     0 DNAT       6    --  !br-mailcow *       0.0.0.0/0            127.0.0.1            tcp dpt:19991 to:172.22.1.250:12345
    7   360 DNAT       6    --  !br-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:4190 to:172.22.1.250:4190
 1983  119K DNAT       6    --  !br-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:995 to:172.22.1.250:995
 3376  194K DNAT       6    --  !br-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:993 to:172.22.1.250:993
  368 21868 DNAT       6    --  !br-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:143 to:172.22.1.250:143
  366 21776 DNAT       6    --  !br-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:110 to:172.22.1.250:110
  933 55132 DNAT       6    --  !br-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:587 to:172.22.1.253:587
19203 1152K DNAT       6    --  !br-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:465 to:172.22.1.253:465
13033  777K DNAT       6    --  !br-mailcow *       0.0.0.0/0            0.0.0.0/0            tcp dpt:25 to:172.22.1.253:25
    0     0 DNAT       6    --  !br-mailcow *       0.0.0.0/0            127.0.0.1            tcp dpt:7654 to:172.22.1.249:6379

Logs of ip6tables -L -vn -t nat:

# Warning: ip6tables-legacy tables present, use ip6tables-legacy to see them
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
1273K   94M DOCKER     0    --  *      *       ::/0                 ::/0                 ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
49138 3931K DOCKER     0    --  *      *       ::/0                !::1                  ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
1686K  164M MASQUERADE  0    --  *      !br-mailcow  fd4d:6169:6c63:6f77::/64  ::/0
    0     0 MASQUERADE  0    --  *      !docker0  fd00:dead:beef:c0::/80  ::/0
    0     0 MASQUERADE  6    --  *      *       fd4d:6169:6c63:6f77::b  fd4d:6169:6c63:6f77::b  tcp dpt:4190
    0     0 MASQUERADE  6    --  *      *       fd4d:6169:6c63:6f77::b  fd4d:6169:6c63:6f77::b  tcp dpt:995
    0     0 MASQUERADE  6    --  *      *       fd4d:6169:6c63:6f77::b  fd4d:6169:6c63:6f77::b  tcp dpt:993
    0     0 MASQUERADE  6    --  *      *       fd4d:6169:6c63:6f77::b  fd4d:6169:6c63:6f77::b  tcp dpt:143
    0     0 MASQUERADE  6    --  *      *       fd4d:6169:6c63:6f77::b  fd4d:6169:6c63:6f77::b  tcp dpt:110
    0     0 MASQUERADE  6    --  *      *       fd4d:6169:6c63:6f77::10  fd4d:6169:6c63:6f77::10  tcp dpt:587
    0     0 MASQUERADE  6    --  *      *       fd4d:6169:6c63:6f77::10  fd4d:6169:6c63:6f77::10  tcp dpt:465
    0     0 MASQUERADE  6    --  *      *       fd4d:6169:6c63:6f77::10  fd4d:6169:6c63:6f77::10  tcp dpt:25

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination
10740  859K RETURN     0    --  br-mailcow *       ::/0                 ::/0
    0     0 RETURN     0    --  docker0 *       ::/0                 ::/0
    1    64 DNAT       6    --  !br-mailcow *       ::/0                 ::/0                 tcp dpt:4190 to:[fd4d:6169:6c63:6f77::b]:4190
   87  6944 DNAT       6    --  !br-mailcow *       ::/0                 ::/0                 tcp dpt:995 to:[fd4d:6169:6c63:6f77::b]:995
 6025  484K DNAT       6    --  !br-mailcow *       ::/0                 ::/0                 tcp dpt:993 to:[fd4d:6169:6c63:6f77::b]:993
  187 15788 DNAT       6    --  !br-mailcow *       ::/0                 ::/0                 tcp dpt:143 to:[fd4d:6169:6c63:6f77::b]:143
   21  1512 DNAT       6    --  !br-mailcow *       ::/0                 ::/0                 tcp dpt:110 to:[fd4d:6169:6c63:6f77::b]:110
    7   544 DNAT       6    --  !br-mailcow *       ::/0                 ::/0                 tcp dpt:587 to:[fd4d:6169:6c63:6f77::10]:587
  259 19096 DNAT       6    --  !br-mailcow *       ::/0                 ::/0                 tcp dpt:465 to:[fd4d:6169:6c63:6f77::10]:465
  453 35196 DNAT       6    --  !br-mailcow *       ::/0                 ::/0                 tcp dpt:25 to:[fd4d:6169:6c63:6f77::10]:25

DNS check:

172.64.155.249
104.18.32.7
@Korsani Korsani added the bug label Apr 8, 2024
@MAGICCC
Copy link
Member

MAGICCC commented Apr 8, 2024

Huh why did you add your domain in the shellscript?

@Korsani
Copy link
Author

Korsani commented Apr 8, 2024

Huh why did you add your domain in the shellscript?

Running freely, the script attempt to renew bunch of certificates (by fetching the domains in mysql db), among them, for example, louisegoutheraud.fr. When renewing this domain, the script makes a directory called autodiscover.louisegoutheraud.fr. That is fine : on next round the directory is still there.

That is NOT the case with bl-evolution.com, as I described : it creates the directory autoconfig.bl-evolution.com (whereas every other domains end by creating autodiscover). To try to track down what happen, I add the domain in the script so that running with bash -x only show what happen with that (guilty) domains, and not with the good ones.

My guess is that the line 337 DOMAINS=${VALIDATED_DOMAINS_SORTED[@]} /srv/obtain-certificate.sh rsa make a wrong directory (autoconfiginstead of autodiscover), because ${VALIDATED_DOMAINS_SORTED[@]} contains wrong informations because line 322 VALIDATED_DOMAINS_SORTED=(${VALIDATED_DOMAINS_ARR[0]} $(echo ${VALIDATED_DOMAINS_ARR[@]:1} | xargs -n1 | sort -u | xargs)) contains mails.<domain> autoconfig.<domain> autodiscover.<domain> for bl-evolution.com and autoconfig.<domain> autodiscover.<domain> for others. Why /srv/obtain-certificate.sh then create autoconfig and not autodiscover ? idk.

So, next round assume that the directory autoconfig is in orphaned directory (why ? i didn't investigate, it was really late :) ) so it deletes it.

Workaround it to symlink autoconfig.bl-evolution.com to autodiscover.bl-evolution.com. So that the reverse-proxy nginx finds its autodiscover.bl-evolution.com

@Korsani
Copy link
Author

Korsani commented Apr 8, 2024

Mmm... I realize that maybe my case is badly named. The nginx reverse-proxy expects certificates in autodiscover. Which, as I mentioned, do not happen with bl-evolution.com : certificate is created in autoconfig.
Still, there is a mismatch between what nginx conf expect, and what /srv/acme.sh does. I'll investigate on our side to see where the nginx conf comes from.

@GamingForLive
Copy link

I dont really know if this related but when i add a fdqn in the acme-mailcow config as an aditional san (as follows):
ADDITIONAL_SAN=smtp.*,myfdqn.de* And restarting acme-mailcow manually it gets the certificate but after a while (about 1 day) it is gone again.

Acme also dosent generate a new one (yes i did run docker compose up -d)
and after looking at https://crt.sh/ i can confirm that no new ssl cert has been issued

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@MAGICCC @Korsani @GamingForLive