Push notification problem with pubsub.chatsecure.cat - prosody 0.11.9 (SASL EXTERNAL failed)

[darklem]

Hi everybody,

After many attempts and a deep search on similar problem, I would really appreciate some help.
I am running an XMPP server on prosody (v0.11.9) and my clients on chatsecure app never received offline push notification.
Based on the server log, the push notifications are well activated :
Push notifications enabled for xxx@MY_DOMAIN.cat/chatsecure79578 (pubsub.chatsecure.org)

But when a message is sent to account connected to the chatsecure app (and the app is closed), I have :

Oct 02 14:55:57 MY_DOMAIN.cat:cloud_notify debug Invoking cloud handle_notify_request() for offline stanza Oct 02 14:55:57 MY_DOMAIN.cat:cloud_notify debug Sending important push notification for nicolas@MY_DOMAIN.cat to pubsub.chatsecure.org (A52799A4-EA42-4F4D-A818-C9C7388399EF) Oct 02 14:55:57 s2sout55e892bfa4f0 debug First attempt to connect to pubsub.chatsecure.org, starting with SRV lookup... Oct 02 14:55:57 adns debug Records for _xmpp-server._tcp.pubsub.chatsecure.org. not in cache, sending query (thread: 0x55e892d0f8b0)... Oct 02 14:55:57 MY_DOMAIN.cat:cloud_notify debug Sending important push notification for nicolas@MY_DOMAIN.cat to pubsub.chatsecure.org (5D6D0D0D-210C-4F77-8ECC-8C44CF52BA51) Oct 02 14:55:57 s2sout55e892bfa4f0 debug trying to send over unauthed s2sout to pubsub.chatsecure.org Oct 02 14:55:57 adns debug Reply for _xmpp-server._tcp.pubsub.chatsecure.org. (thread: 0x55e892d0f8b0) Oct 02 14:55:57 s2sout55e892bfa4f0 debug pubsub.chatsecure.org has SRV records, handling... Oct 02 14:55:57 s2sout55e892bfa4f0 debug Best record found, will connect to pubsub.chatsecure.org.:5269 Oct 02 14:55:57 adns debug Records for pubsub.chatsecure.org. not in cache, sending query (thread: 0x55e892c61390)... Oct 02 14:55:57 adns debug Reply for pubsub.chatsecure.org. (thread: 0x55e892c61390) Oct 02 14:55:57 s2sout55e892bfa4f0 debug DNS reply for pubsub.chatsecure.org. gives us 45.55.5.246 Oct 02 14:55:57 s2sout55e892bfa4f0 debug Beginning new connection attempt to pubsub.chatsecure.org ([45.55.5.246]:5269) Oct 02 14:55:58 s2sout55e892bfa4f0 debug Sending[s2sout_unauthed]: <stream:stream to='pubsub.chatsecure.org' xml:lang='en' version='1.0' xmlns:db='jabber:server:dialback' xmlns:stream='http://etherx.jabber.org/streams' xmlns='jabber:server' from='MY_DOMAIN.cat'> Oct 02 14:55:58 MY_DOMAIN.cat:tls debug pubsub.chatsecure.org is offering TLS, taking up the offer... Oct 02 14:55:59 s2sout55e892bfa4f0 debug Sending[s2sout_unauthed]: <stream:stream to='pubsub.chatsecure.org' xml:lang='en' version='1.0' xmlns:db='jabber:server:dialback' xmlns:stream='http://etherx.jabber.org/streams' xmlns='jabber:server' from='MY_DOMAIN.cat'> Oct 02 14:55:59 x509 debug Cert dNSName pubsub.chatsecure.org matched hostname Oct 02 14:55:59 MY_DOMAIN.cat:saslauth debug Initiating SASL EXTERNAL with pubsub.chatsecure.org Oct 02 14:55:59 MY_DOMAIN.cat:saslauth debug SASL EXTERNAL failed, falling back to dialback Oct 02 14:55:59 s2sout55e892bfa4f0 debug Sending[s2sout_unauthed]: <db:result to='pubsub.chatsecure.org' from='MY_DOMAIN.cat'> Oct 02 14:55:59 socket debug server.lua: client 45.55.5.246:clientport read error: closed Oct 02 14:55:59 s2sout55e892bfa4f0 debug s2s disconnected: MY_DOMAIN.cat->pubsub.chatsecure.org (closed) Oct 02 14:55:59 s2sout55e892bfa4f0 debug Destroying outgoing session MY_DOMAIN.cat->pubsub.chatsecure.org: closed Oct 02 14:55:59 s2sout55e892bfa4f0 info Sending error replies for 2 queued stanzas because of failed outgoing connection to pubsub.chatsecure.org Oct 02 14:55:59 stanzarouter debug Received[s2sin]: <iq to='MY_DOMAIN.cat' type='error' id='2c99f7318acfd37ab3f02abc4bdfe1ea6dc5b5075d9199750a97a4829bf6ede8' from='pubsub.chatsecure.org'> Oct 02 14:55:59 MY_DOMAIN.cat:cloud_notify info Got error of type 'cancel' (remote-server-not-found) for identifier 'pubsub.chatsecure.org<A52799A4-EA42-4F4D-A818-C9C7388399EF': error count for this identifier is now at 1 Oct 02 14:55:59 stanzarouter debug Received[s2sin]: <iq to='MY_DOMAIN.cat' type='error' id='9b4151632d0ff06bc73bb4abfe27456b6f4fa61ff129d78b36ce27755dd710b0' from='pubsub.chatsecure.org'> Oct 02 14:55:59 MY_DOMAIN.cat:cloud_notify info Got error of type 'cancel' (remote-server-not-found) for identifier 'pubsub.chatsecure.org<5D6D0D0D-210C-4F77-8ECC-8C44CF52BA51': error count for this identifier is now at 3

I use Let's Encrypt certificates and s2s_secure_auth is true.
In addition, I managed to activate the SASL authebtification when doing the same with pubsub.tigase.org

Thank you very much for the future help
Regards
DL

[gertyq]

I had the same problem after Sep 30. It seems that the problem is with Let's Encrypt root cert. You should use certificate with alternative chain. Via certbot with option --preferred-chain "ISRG Root X1"

[mettKK]

HI, same problem as DL here.
Server OS upgraded last Sat(10/16) (Not using Let's Encrypted DST Root CA X3 certificate which was valid until Sep 30).

[chrisballinger]

Thank you for the report! How about now?

[mettKK]

Hi,
Thank you for your answer.

I just tried again and no change:
-with encryption and auth force
-with encryption force
-without encryption and auth force.

Do not know if that applies but if you are using an old Debian
you need to disable Let's Encrypted "DST Root CA X3" from the store
as follows:
vi /etc/ca-certificates
#add an ! front of the cert:
!mozilla/DST_Root_CA_X3.crt
#save;
#then rebuild the store with
update-ca-certificates --fresh

[mettKK]

Hi,

Here is a log of what is happening
https://conference.pmars.jp:5281/pastebin/3bb56a3a-16f9-4a12-bb0b-a5ba827a1aa2

TIA,

[chrisballinger]

Ah I see, I initially thought the problem was the certs on pubsub.chatsecure.org needed to be regenerated, but it looks like it's the client side validation that's failing.

e.g. https://superuser.com/questions/1679204/curl-on-ubuntu-14-all-lets-encrypt-certificates-are-expired-error-60

I just updated the cert store and disabled DST Root CA X3.

How about now?

[schorschii]

Hi @chrisballinger ,
Thanks for your response. Unfortunately, nothing changed. As described in #1251 , when I include the valid "ISRG Root X1" root CA (cross signed by the expired "DST Root CA X3") into my server certificate used by ejabberd, I still get:

2021-10-24 21:33:45.097 [warning] <0.544.0>@ejabberd_s2s_out:handle_auth_failure:226 (tls|<0.544.0>) Failed outbound s2s EXTERNAL authentication sieber.systems -> pubsub.chatsecure.org (45.55.5.246): Authentication failed: Peer responded with error: not-authorized
2021-10-24 21:33:45.097 [warning] <0.544.0>@ejabberd_s2s_out:process_auth_result:141 Failed to establish outbound s2s connection sieber.systems -> pubsub.chatsecure.org: authentication failed; bouncing for 164 seconds

When I remove the CA certificate from my server certificate file, there is no error in the log anymore, even a success message is visible:

2021-10-24 21:28:57.182 [info] <0.569.0>@ejabberd_s2s_out:init:280 Outbound s2s connection started: sieber.systems -> pubsub.chatsecure.org
2021-10-24 21:28:58.492 [info] <0.569.0>@ejabberd_s2s_out:handle_auth_success:216 (tls|<0.569.0>) Accepted outbound s2s EXTERNAL authentication sieber.systems -> pubsub.chatsecure.org (45.55.5.246)
2021-10-24 21:29:00.291 [info] <0.572.0>@ejabberd_s2s_in:handle_auth_success:181 (tls|<0.572.0>) Accepted inbound s2s EXTERNAL authentication pubsub.chatsecure.org -> sieber.systems (::ffff:45.55.5.246)

But the push notifications are still not working (while they do in other XMPP apps for iOS).

[darklem]

Hi @chrisballinger
Thank you for your answer. I confirm the same situation with prosody described by @schorschii.

[licaon-kter]

Did you try turning it off and on?

[gertyq]

Ah I see, I initially thought the problem was the certs on pubsub.chatsecure.org needed to be regenerated, but it looks like it's the client side validation that's failing.

e.g. https://superuser.com/questions/1679204/curl-on-ubuntu-14-all-lets-encrypt-certificates-are-expired-error-60

I just updated the cert store and disabled DST Root CA X3.

How about now?

What you mean clientside ? ChatSecure app or iOS itself or XMPP server (that is some kind of client during SASL auth). According to expiring DST Root CA X3 cert all libs (like OpenSSL <1.1.0) that used them in trust chains now will fail. But also all endpoints that don't trust ISRG Root X1 cert without DST Root CA X3 cert in chain will fail with endpoints that uses only ISRG Root X1 cert in chain. So the only right solution is ignore DST Root CA X3 cert and trust ISRG Root X1 cert in all places where it possible.

[chrisballinger]

Oh I was thinking more like, the pubsub server's CA store rejecting the LE cert issued by Heroku's ACM setup (which hosts push.chatsecure.org). Apologies for the delay, I was hoping this was going to be a quicker fix. Will have to do some more digging.

[netboy3]

Any update on this issue? pubsub.chatsecure.org still seems to SASL reject connections:

xxx.yyy.zzz:saslauth      info    SASL EXTERNAL with pubsub.chatsecure.org failed: error<cancel:not-authorized:>

As others indicated, it works fine with Tigase and Monal.

[gdt]

I am seeing failures with ejabberd.

2022-10-04 13:24:17.554355-04:00 [warning] <0.2446.0>@ejabberd_s2s_out:handle_auth_failure/3:233 (tls|<0.2446.0>) Failed outbound s2s EXTERNAL authentication j.example.com -> pubsub.chatsecure.org (45.55.5.246): Authentication failed: Peer responded with error: not-authorized

The inbound connection succeeds, and I have 2-way peering with a number of other domains. Also connections to push.tigase.im work, both in the logs and I hear the client beep promptly. chatsecure does not get messages.

[gdt]

Time is EDT if you want to look it up, so 172417 UTC.

[licaon-kter]

@gdt if you still have ChatSecure users please guide them to install either Monal (https://eversten.net/en/blog/monal/) or Siskin/Snikket (https://eversten.net/en/blog/siskin/) pls

[gdt]

Thanks. I will do that. I take your comment as a clue that the Chatsecure code/project is no longer really maintained. I had already tried siskin and found it to work. Thanks also for the eversten links.

[chrisballinger]

Yep, unfortunately it is in critical maintenance-only mode.

[licaon-kter]

@gdt ah, also read https://eversten.net/en/blog/notification/