Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Federation tester says okay but federation and mail sending fails - OpenSSL/TLS cert CA issue #10976

Closed
vmario89 opened this issue Oct 2, 2021 · 13 comments
Closed

Federation tester says okay but federation and mail sending fails - OpenSSL/TLS cert CA issue #10976

vmario89 opened this issue Oct 2, 2021 · 13 comments

Comments

@vmario89
Copy link

vmario89 commented Oct 2, 2021

Hi,
i have serious issue and i dont know how to fix. i invested a lot of time but get no clue about.
https://federationtester.matrix.org/api/report?server_name=matrix.fablabchemnitz.de says our server is fine, but i cannot federate from external matrix servers.
the users on same instance can chat properly.
there is some problem with TLS/SSL and certificate but the cert is fine so far, i think. We use Lets Encrypt

i already posted at closed issue but i started this as a new one because i think its a different issue

related issues are:

postfix log:

  • SSL3 alert read:fatal:unknown CA
  • Okt 02 00:17:05 postfix/smtpd[3054606]: SSL_accept:error in error
  • Okt 02 00:17:05 postfix/smtpd[3054606]: SSL_accept error from localhost[127.0.0.1]: -1
  • Okt 02 00:17:05 postfix/smtpd[3054606]: warning: TLS library problem: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca:../ssl/record/rec_layer_s3.c:1543:SSL alert number 48:
  • Okt 02 00:17:05 postfix/smtpd[3054606]: lost connection after STARTTLS from localhost[127.0.0.1]

the log /var/log/matrix-synapse/homeserver.log says issue about sending mails


2021-10-02 00:49:25,542 - twisted - 271 - INFO - sentinel - SMTP Client retrying server. Retry: 5
2021-10-02 00:49:25,562 - twisted - 271 - INFO - sentinel - SMTP Client retrying server. Retry: 4
2021-10-02 00:49:25,582 - twisted - 271 - INFO - sentinel - SMTP Client retrying server. Retry: 3
2021-10-02 00:49:25,593 - twisted - 271 - INFO - sentinel - SMTP Client retrying server. Retry: 2
2021-10-02 00:49:25,613 - twisted - 271 - INFO - sentinel - SMTP Client retrying server. Retry: 1
2021-10-02 00:49:25,733 - synapse.handlers.identity - 415 - ERROR - POST-145 - Error sending threepid validation email to <redacted>@<redacted>.de
Traceback (most recent call last):
  File "/opt/venvs/matrix-synapse/lib/python3.8/site-packages/synapse/handlers/identity.py", line 413, in send_threepid_validation
    await send_email_func(email_address, token, client_secret, session_id)
  File "/opt/venvs/matrix-synapse/lib/python3.8/site-packages/synapse/push/mailer.py", line 201, in send_add_threepid_mail
    await self.send_email(
  File "/opt/venvs/matrix-synapse/lib/python3.8/site-packages/synapse/push/mailer.py", line 318, in send_email
    await self.send_email_handler.send_email(
  File "/opt/venvs/matrix-synapse/lib/python3.8/site-packages/synapse/handlers/send_email.py", line 175, in send_email
    await self._sendmail(
  File "/opt/venvs/matrix-synapse/lib/python3.8/site-packages/synapse/handlers/send_email.py", line 111, in _sendmail
    await make_deferred_yieldable(d)
twisted.mail._except.SMTPConnectError: Unable to connect to server.
2021-10-02 00:49:25,734 - synapse.http.server - 88 - INFO - POST-145 - <XForwardedForRequest at 0x7fb2743e2130 method='POST' uri='/_matrix/client/r0/account/3pid/email/requestToken' clientproto='HTTP/1.0' site='8008'> SynapseError: 500 - An error was encountered when sending the email

and it also says about external room joins:

2021-10-02 11:15:18,162 - synapse.crypto.keyring - 633 - INFO - PerspectivesKeyFetcher-24 - Requesting keys [_FetchKeyRequest(server_name='matrix.org', minimum_valid_until_ts=1633166118161, key_ids=['ed25519:a_RXGa'])] from notary server matrix.org
2021-10-02 11:15:18,162 - synapse.crypto.keyring - 588 - WARNING - PerspectivesKeyFetcher-24 - Key lookup failed from 'matrix.org': Not retrying server matrix.org.
2021-10-02 11:15:18,193 - synapse.http.matrixfederationclient - 635 - INFO - ServerKeyFetcher-24 - {GET-O-54} [matrix.org] Request failed: GET matrix://matrix.org/_matrix/key/v2/server/ed25519%3Aa_RXGa: ResponseNeverReceived:[Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])]
2021-10-02 11:15:18,193 - synapse.crypto.keyring - 786 - WARNING - ServerKeyFetcher-24 - Error looking up keys ['ed25519:a_RXGa'] from matrix.org: Failed to send request: ResponseNeverReceived: [<twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')]>]
2021-10-02 11:15:18,193 - synapse.federation.transport.server._base - 267 - WARNING - GET-1222 - authenticate_request failed: 401: Failed to find any key to satisfy: _FetchKeyRequest(server_name='matrix.org', minimum_valid_until_ts=1633166118161, key_ids=['ed25519:a_RXGa'])
2021-10-02 11:15:18,193 - synapse.http.server - 88 - INFO - GET-1222 - <XForwardedForRequest at 0x7f5db0102b50 method='GET' uri='/_matrix/federation/v1/query/profile?user_id=%40vmario89%3Amatrix.fablabchemnitz.de&field=displayname' clientproto='HTTP/1.1' site='8448'> SynapseError: 401 - Failed to find any key to satisfy: _FetchKeyRequest(server_name='matrix.org', minimum_valid_until_ts=1633166118161, key_ids=['ed25519:a_RXGa'])
2021-10-02 11:15:18,194 - synapse.access.https.8448 - 389 - INFO - GET-1222 - 176.126.240.158 - 8448 - {None} Processed request: 0.033sec/-0.000sec (0.000sec, 0.000sec) (0.000sec/0.000sec/0) 182B 401 "GET /_matrix/federation/v1/query/profile?user_id=%40vmario89%3Amatrix.fablabchemnitz.de&field=displayname HTTP/1.1" "Synapse/1.44.0rc1 (b=matrix-org-hotfixes,4cb6ffdf4)" [0 dbevts]
2021-10-02 11:15:18,374 - synapse.crypto.keyring - 633 - INFO - PerspectivesKeyFetcher-25 - Requesting keys [_FetchKeyRequest(server_name='matrix.org', minimum_valid_until_ts=1633166118373, key_ids=['ed25519:a_RXGa'])] from notary server matrix.org
2021-10-02 11:15:18,374 - synapse.crypto.keyring - 588 - WARNING - PerspectivesKeyFetcher-25 - Key lookup failed from 'matrix.org': Not retrying server matrix.org.
2021-10-02 11:15:18,398 - synapse.http.matrixfederationclient - 635 - INFO - ServerKeyFetcher-25 - {GET-O-56} [matrix.org] Request failed: GET matrix://matrix.org/_matrix/key/v2/server/ed25519%3Aa_RXGa: ResponseNeverReceived:[Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])]
2021-10-02 11:15:18,398 - synapse.crypto.keyring - 786 - WARNING - ServerKeyFetcher-25 - Error looking up keys ['ed25519:a_RXGa'] from matrix.org: Failed to send request: ResponseNeverReceived: [<twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')]>]
2021-10-02 11:15:18,398 - synapse.federation.transport.server._base - 267 - WARNING - PUT-1223 - authenticate_request failed: 401: Failed to find any key to satisfy: _FetchKeyRequest(server_name='matrix.org', minimum_valid_until_ts=1633166118373, key_ids=['ed25519:a_RXGa'])

i checked CA cert which is fine. All things with mail clients work properly, like thunderbird or roundcube or other services which connect to the mail server from exernal or localhost

my mail server only supports TLS 1.2 and TLS 1.3. SSLv2/v3 is completely disabled for smtp/smtpd. I already tried to tie down to TLS 1.0

i updated twisted[tls] und removed the older apt package too python3-twisted

also tried synapse connecting to mail server locally or by public mail server address

some infos about versions:

  • ii openssl 1.1.1f-1ubuntu2.8 amd64 Secure Sockets Layer toolkit - cryptographic utility
  • ii postfix 3.4.13-0ubuntu1.2 amd64 High-performance mail transport agent
  • ii matrix-synapse-py3 1.43.0+focal1 amd64 Open federated Instant Messaging and VoIP server
  • twisted 21.7.0
  • recent version of 21.5.0

some more stuff i tried which also looks good, but without solving or modifying the problem:

update-ca-certificates

openssl verify -CAfile /etc/ssl/certs/ISRG_Root_X1.pem /etc/letsencrypt/live/mymailserver.de/chain.pem

openssl s_client -showcerts -servername mail.mymailserver.org -connect smtp.mymailserver.org:587
openssl s_client -starttls smtp -showcerts -servername mymailserver.org -connect smtp.mymailserver.org:587
openssl s_client -CApath /etc/ssl/certs -CAfile /etc/ssl/certs/ca-certificates.crt -starttls smtp -showcerts -servername mail.mymailserver.org -connect smtp.mymailserver.org:587
openssl s_client -CAfile /etc/ssl/certs/ca-certificates.crt -starttls smtp -showcerts -servername mail.mymailserver.org -connect mail.mymailserver.org:25

apt remove python3-openssl
apt remove python3-twisted
pip3 install --upgrade twisted[tls]
pip3 install --upgrade treq
pip3 install --upgrade cryptography
pip3 install --upgrade secretstorage
pip3 install --upgrade pyOpenSSL


openssl s_client -connect matrix.fablabchemnitz.de:443

gives "Verify return code: 2 (unable to get issuer certificate)"
@davehayes
Copy link

Does this possibly have to do with the Let's Encrypt root certificate expiring?

@vmario89
Copy link
Author

vmario89 commented Oct 4, 2021 •
edited

Hi. Maybe. Letsencrypt has new root cert ISRG root X1. It expired at 30.09. But i already renewed all certs and restarted all services/servers mulitple times.

@babolivier
Copy link
Contributor

This sounds like there's an issue with your setup rather than a bug in Synapse. We only use GitHub issues for bug reports, not for support requests. For support requests, please use the room #synapse:matrix.org on Matrix.

@hasanihunter
Copy link

Hi @babolivier,

From the output it looks like the root certificate did expire

echo quit | openssl s_client -connect matrix.org:443

CONNECTED(00000003)
depth=3 O = Digital Signature Trust Co., CN = DST Root CA X3
verify error:num=10:certificate has expired
notAfter=Sep 30 14:01:15 2021 GMT
---
Certificate chain
 0 s:/CN=www.matrix.org
   i:/C=US/O=Let's Encrypt/CN=R3
 1 s:/C=US/O=Let's Encrypt/CN=R3
   i:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
 2 s:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
   i:/O=Digital Signature Trust Co./CN=DST Root CA X3
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIG5zCCBc+gAwIBAgISA9fZyQ7dkJuslcQVA1OHuG+QMA0GCSqGSIb3DQEBCwUA
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
EwJSMzAeFw0yMTA5MDIwODMzMDZaFw0yMTEyMDEwODMzMDVaMBkxFzAVBgNVBAMT
Dnd3dy5tYXRyaXgub3JnMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA
xVkNlBjdnI7aTcJFoMLSNoDCzKOx4DZRhLp92TvfiwtXBGP4XzzsExdsameeZcI6
eNEhCP7Cy8xTe9uTBxOmq06Uvf2XbJS2rltij7g+588a6ps7eaw7NJtMPRbpYrSh
sTYjQQXKl0E/btwacnlLft1DX0q2At+zqhhcRsd8+7iUpzTrU0arWkgKUW5gvmW6
GeHL9/BL9Yfyc5bH6K6ByvhiOC98wOcdyIxlWLM9XtPVcORQy0ms0HNSx6pN5caw
CtxcDIXcONucccJ+b2P7YXnplGc5qdM4yO9hOd3G5E+DYL7moJLyFs/wIYlpBw3S
4Kvs84Spl6Z4iQVGuWSt6ToQ4DBcSjMLMzenR1zelXiYfYKAYiBZOStETXOLtQNM
sRzw+YAStg13Xnk4o9oZXYKAHNfYsnFllj3eKkvT8otjNKhpDuByIsvwuL6l3KR3
87hXDboP5y3dThAQXtlnJVTradaB+Ffp9P2zRcG81v0hyNN3k0krygJV8ZIRUvCF
2dfuc+Wzl/aMIf0ZUohouk5zSid9Cvclb7e+v4I2dpkAky3MtnZp20IpKvk8857f
tzPEuYCR0mLhK12OzX8K5rGlNfcecfg/PQAlRr+cGWCkmTWIOzt3Tws/LPAN+y8k
Lsrj4M9Zq70l+edsWZUFLpT+KVCF3ap7R91bLzkIa5cCAwEAAaOCAw4wggMKMA4G
A1UdDwEB/wQEAwIFoDAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDAYD
VR0TAQH/BAIwADAdBgNVHQ4EFgQU60h02s4Bdg4lJtWSZe4fZzAsBRUwHwYDVR0j
BBgwFoAUFC6zF7dYVsuuUAlA5h+vnYsUwsYwVQYIKwYBBQUHAQEESTBHMCEGCCsG
AQUFBzABhhVodHRwOi8vcjMuby5sZW5jci5vcmcwIgYIKwYBBQUHMAKGFmh0dHA6
Ly9yMy5pLmxlbmNyLm9yZy8wgd0GA1UdEQSB1TCB0oINYWJvdXQucmlvdC5pbYIR
YXJld2VyZWFkeXlldC5jb22CDmxmcy5tYXRyaXgub3JnghhtYXRyaXgtY2xpZW50
Lm1hdHJpeC5vcmeCHG1hdHJpeC1mZWRlcmF0aW9uLm1hdHJpeC5vcmeCCm1hdHJp
eC5vcmeCCW1hdHJpeC50b4IKbW9kdWxhci5pbYIHcmlvdC5pbYIPc3BlYy5tYXRy
aXgub3Jnggl2ZWN0b3IuaW2CDnd3dy5tYXRyaXgub3Jngg53d3cubW9kdWxhci5p
bTBMBgNVHSAERTBDMAgGBmeBDAECATA3BgsrBgEEAYLfEwEBATAoMCYGCCsGAQUF
BwIBFhpodHRwOi8vY3BzLmxldHNlbmNyeXB0Lm9yZzCCAQQGCisGAQQB1nkCBAIE
gfUEgfIA8AB2APZclC/RdzAiFFQYCDCUVo7jTRMZM7/fDC8gC8xO8WTjAAABe6Xa
0lEAAAQDAEcwRQIhAO1PYxrVVhp/rKvu27WHONo0nd/AXt1z9vsQ0mg33OqsAiAm
mzM7zaTjFk7sNCYQCokhnraKXd98C6yZug2r2NMZWwB2AG9Tdqwx8DEZ2JkApFEV
/3cVHBHZAsEAKQaNsgiaN9kTAAABe6Xa0ocAAAQDAEcwRQIhAN0mlh7rsSh/7gya
5kCv2iKoVKgdCtd3rQNet47IHivLAiAkVKmqNEh4ZR9gE4tpd9WTUK4wU5gR2lWW
79cChdm7cjANBgkqhkiG9w0BAQsFAAOCAQEAZ+cRZSi1FeHF4meCZmV4pS2iDFaB
zMRns9FnS0zDCS29QrhSYEaJtdnANDi3E5FJqckF+U8Nafohx1BbIQ0tnjFk0zdX
A9L+7SjsgOLUxrAbXT9C0uWvEGbBZ8p1mJMtboKmgV0dLPYsNrGQ/2bK4SYxnyzy
bIyzNGfQEw3HtKRFybrcpHSjLvVMZL2QhhzZ7tpjUpREw++mRV3oBXVwU/SjG38E
9U7RlFQNTzLK2paZB8wEqWMMo6xt0Ot45oaJrGiIwUSM9ql3+19h1DP8RaQYpVV8
JDMmXios7EANq7pgvwPQ8qqCNRSB8+pqGbGfBNjmBc1bf7SlaTfUIjQ/Eg==
-----END CERTIFICATE-----
subject=/CN=www.matrix.org
issuer=/C=US/O=Let's Encrypt/CN=R3
---
No client certificate CA names sent
Peer signing digest: SHA256
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 5387 bytes and written 433 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 4096 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-GCM-SHA256
    Session-ID: 5A458AA6B4AF1CB431DAD316E7B7D534AF3054F3D07E86DF5DC2C3022B9E84C8
    Session-ID-ctx: 
    Master-Key: 89ED5D1E86E7823C2A283BD2EBD725A2EBD41AD35DA06E7D6426D1863F70D6C7364BF7F1AE4F5E688F5CB9C9CFF386D7
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 64800 (seconds)
    TLS session ticket:
    0000 - a3 ea 17 fa 72 94 b9 52-56 06 68 c0 26 4a 4d d4   ....r..RV.h.&JM.
    0010 - 70 ac 9f 78 3e c9 1f 30-18 a7 77 14 8d c8 2f 12   p..x>..0..w.../.
    0020 - 55 9e 0d 0e a0 d7 c4 6f-b4 15 a8 91 47 6a 3a 0b   U......o....Gj:.
    0030 - 2a 37 bc 6c c7 7c 74 21-fd 07 6b 1c 99 a0 36 d4   *7.l.|t!..k...6.
    0040 - 96 08 c8 43 11 ba be 9c-ad 7c a3 16 f8 4b 0e b8   ...C.....|...K..
    0050 - a3 68 48 47 98 0b f6 91-f8 fa 48 01 92 d7 67 fb   .hHG......H...g.
    0060 - 3e de 47 45 99 29 b4 44-10 89 c3 26 e2 43 1a a5   >.GE.).D...&.C..
    0070 - 10 12 a7 f8 55 46 9c 8f-8e a4 73 0f ff 1c 5b 8a   ....UF....s...[.
    0080 - 97 99 02 cb fd af 95 7b-3e 1d 44 a1 59 28 79 a6   .......{>.D.Y(y.
    0090 - ca e3 42 43 1d a0 06 a4-7f 36 3a ef db ff 3a d9   ..BC.....6:...:.
    00a0 - 8f b6 f9 91 9a b7 db 9a-44 ff 8c 69 2a d0 3c 74   ........D..i*.<t

    Start Time: 1633450133
    Timeout   : 300 (sec)
    Verify return code: 10 (certificate has expired)
---
DONE

@hasanihunter
Copy link

This sounds like there's an issue with your setup rather than a bug in Synapse. We only use GitHub issues for bug reports, not for support requests. For support requests, please use the room #synapse:matrix.org on Matrix.

I'm having the same issue as the OP, but I cannot join the #synapse:matrix.org room because my homeserver cannot connect to matrix.org (because of the expired root cert)

@callahad
Copy link
Contributor

callahad commented Oct 6, 2021

@hasanihunter What version of openssl are you using?

This is certainly caused by the DST Root CA X3 Expiration. The default chain recommended by Let's Encrypt retains the expired DST Root CA X3 in order to extend Android device compatibility.

As mentioned in the above links, OpenSSL v1.0.x is incompatible with this chain, even when it includes the ISRG Root X1 in its trust store. Can you upgrade to OpenSSL v1.1.x? Further details at https://community.letsencrypt.org/t/openssl-client-compatibility-changes-for-let-s-encrypt-certificates/143816

@hasanihunter
Copy link

@callahad on a stock M1 mac mini, the version is LibreSSL 2.8.3 and the output on the mac mini is:

CONNECTED(00000005)
depth=1 O = Digital Signature Trust Co., CN = DST Root CA X3
verify error:num=10:certificate has expired
notAfter=Sep 30 14:01:15 2021 GMT
verify return:0
depth=1 O = Digital Signature Trust Co., CN = DST Root CA X3
verify error:num=10:certificate has expired
notAfter=Sep 30 14:01:15 2021 GMT
verify return:0
depth=3 O = Digital Signature Trust Co., CN = DST Root CA X3
verify error:num=10:certificate has expired
notAfter=Sep 30 14:01:15 2021 GMT
verify return:0
---
Certificate chain
 0 s:/CN=www.matrix.org
   i:/C=US/O=Let's Encrypt/CN=R3
 1 s:/C=US/O=Let's Encrypt/CN=R3
   i:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
 2 s:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
   i:/O=Digital Signature Trust Co./CN=DST Root CA X3
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIG5zCCBc+gAwIBAgISA9fZyQ7dkJuslcQVA1OHuG+QMA0GCSqGSIb3DQEBCwUA
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
EwJSMzAeFw0yMTA5MDIwODMzMDZaFw0yMTEyMDEwODMzMDVaMBkxFzAVBgNVBAMT
Dnd3dy5tYXRyaXgub3JnMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA
xVkNlBjdnI7aTcJFoMLSNoDCzKOx4DZRhLp92TvfiwtXBGP4XzzsExdsameeZcI6
eNEhCP7Cy8xTe9uTBxOmq06Uvf2XbJS2rltij7g+588a6ps7eaw7NJtMPRbpYrSh
sTYjQQXKl0E/btwacnlLft1DX0q2At+zqhhcRsd8+7iUpzTrU0arWkgKUW5gvmW6
GeHL9/BL9Yfyc5bH6K6ByvhiOC98wOcdyIxlWLM9XtPVcORQy0ms0HNSx6pN5caw
CtxcDIXcONucccJ+b2P7YXnplGc5qdM4yO9hOd3G5E+DYL7moJLyFs/wIYlpBw3S
4Kvs84Spl6Z4iQVGuWSt6ToQ4DBcSjMLMzenR1zelXiYfYKAYiBZOStETXOLtQNM
sRzw+YAStg13Xnk4o9oZXYKAHNfYsnFllj3eKkvT8otjNKhpDuByIsvwuL6l3KR3
87hXDboP5y3dThAQXtlnJVTradaB+Ffp9P2zRcG81v0hyNN3k0krygJV8ZIRUvCF
2dfuc+Wzl/aMIf0ZUohouk5zSid9Cvclb7e+v4I2dpkAky3MtnZp20IpKvk8857f
tzPEuYCR0mLhK12OzX8K5rGlNfcecfg/PQAlRr+cGWCkmTWIOzt3Tws/LPAN+y8k
Lsrj4M9Zq70l+edsWZUFLpT+KVCF3ap7R91bLzkIa5cCAwEAAaOCAw4wggMKMA4G
A1UdDwEB/wQEAwIFoDAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDAYD
VR0TAQH/BAIwADAdBgNVHQ4EFgQU60h02s4Bdg4lJtWSZe4fZzAsBRUwHwYDVR0j
BBgwFoAUFC6zF7dYVsuuUAlA5h+vnYsUwsYwVQYIKwYBBQUHAQEESTBHMCEGCCsG
AQUFBzABhhVodHRwOi8vcjMuby5sZW5jci5vcmcwIgYIKwYBBQUHMAKGFmh0dHA6
Ly9yMy5pLmxlbmNyLm9yZy8wgd0GA1UdEQSB1TCB0oINYWJvdXQucmlvdC5pbYIR
YXJld2VyZWFkeXlldC5jb22CDmxmcy5tYXRyaXgub3JnghhtYXRyaXgtY2xpZW50
Lm1hdHJpeC5vcmeCHG1hdHJpeC1mZWRlcmF0aW9uLm1hdHJpeC5vcmeCCm1hdHJp
eC5vcmeCCW1hdHJpeC50b4IKbW9kdWxhci5pbYIHcmlvdC5pbYIPc3BlYy5tYXRy
aXgub3Jnggl2ZWN0b3IuaW2CDnd3dy5tYXRyaXgub3Jngg53d3cubW9kdWxhci5p
bTBMBgNVHSAERTBDMAgGBmeBDAECATA3BgsrBgEEAYLfEwEBATAoMCYGCCsGAQUF
BwIBFhpodHRwOi8vY3BzLmxldHNlbmNyeXB0Lm9yZzCCAQQGCisGAQQB1nkCBAIE
gfUEgfIA8AB2APZclC/RdzAiFFQYCDCUVo7jTRMZM7/fDC8gC8xO8WTjAAABe6Xa
0lEAAAQDAEcwRQIhAO1PYxrVVhp/rKvu27WHONo0nd/AXt1z9vsQ0mg33OqsAiAm
mzM7zaTjFk7sNCYQCokhnraKXd98C6yZug2r2NMZWwB2AG9Tdqwx8DEZ2JkApFEV
/3cVHBHZAsEAKQaNsgiaN9kTAAABe6Xa0ocAAAQDAEcwRQIhAN0mlh7rsSh/7gya
5kCv2iKoVKgdCtd3rQNet47IHivLAiAkVKmqNEh4ZR9gE4tpd9WTUK4wU5gR2lWW
79cChdm7cjANBgkqhkiG9w0BAQsFAAOCAQEAZ+cRZSi1FeHF4meCZmV4pS2iDFaB
zMRns9FnS0zDCS29QrhSYEaJtdnANDi3E5FJqckF+U8Nafohx1BbIQ0tnjFk0zdX
A9L+7SjsgOLUxrAbXT9C0uWvEGbBZ8p1mJMtboKmgV0dLPYsNrGQ/2bK4SYxnyzy
bIyzNGfQEw3HtKRFybrcpHSjLvVMZL2QhhzZ7tpjUpREw++mRV3oBXVwU/SjG38E
9U7RlFQNTzLK2paZB8wEqWMMo6xt0Ot45oaJrGiIwUSM9ql3+19h1DP8RaQYpVV8
JDMmXios7EANq7pgvwPQ8qqCNRSB8+pqGbGfBNjmBc1bf7SlaTfUIjQ/Eg==
-----END CERTIFICATE-----
subject=/CN=www.matrix.org
issuer=/C=US/O=Let's Encrypt/CN=R3
---
No client certificate CA names sent
Server Temp Key: ECDH, X25519, 253 bits
---
SSL handshake has read 5346 bytes and written 281 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-CHACHA20-POLY1305
Server public key is 4096 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-CHACHA20-POLY1305
    Session-ID: 7C19C89BF60D580317E171730E004E9ECC87900A9B1B13009E5C4B6FEAB4EA99
    Session-ID-ctx: 
    Master-Key: 52D5AB62F47130646A5E1B563512846B739036ECC7CC408B8607F90D2D48B73BD430ED8E54D14F75B2E085010F53EFF6
    TLS session ticket lifetime hint: 64799 (seconds)
    TLS session ticket:
    0000 - 42 fd ed 6a bf 52 30 e4-f8 3b 7a d0 43 68 8a b0   B..j.R0..;z.Ch..
    0010 - a4 60 2d 88 fd b2 67 a8-df 1e ad 39 98 2a 8d 2e   .`-...g....9.*..
    0020 - e5 8c 1d 16 49 8c be 24-07 ad 8c 3b b9 0e ed 84   ....I..$...;....
    0030 - e9 31 a7 40 4a 1a fa 91-86 3d f4 6c 78 bb 63 bf   .1.@J....=.lx.c.
    0040 - ba 7b aa 59 67 3c cd b3-07 af 0b 3a 90 4f de d7   .{.Yg<.....:.O..
    0050 - c0 fe 2a 92 eb 3f f6 08-e7 f3 ab c2 72 79 95 39   ..*..?......ry.9
    0060 - 03 21 27 98 08 27 9a fd-29 0e 0a de aa 60 7e 93   .!'..'..)....`~.
    0070 - 4d 79 7b ed 43 53 f7 f7-05 d4 f6 5b 09 53 8f 89   My{.CS.....[.S..
    0080 - 66 11 26 d6 56 3b 09 80-77 f6 cd 15 d5 92 c8 49   f.&.V;..w......I
    0090 - e9 6d 47 a3 a5 5f a5 3c-2f aa 5d 2d 45 e7 3d f6   .mG.._.</.]-E.=.
    00a0 - 60 da 15 c9 0b 20 6f 5b-12 2d 9b e2 29 0c 14 79   `.... o[.-..)..y

    Start Time: 1633533276
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
---
DONE

On my FreeBSD server, I upgraded to FreeBSD v12.2 and I have OpenSSL 1.1.1 with the output:

CONNECTED(00000003)
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = www.matrix.org
verify return:1
---
Certificate chain
 0 s:CN = www.matrix.org
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
 2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   i:O = Digital Signature Trust Co., CN = DST Root CA X3
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIG5zCCBc+gAwIBAgISA9fZyQ7dkJuslcQVA1OHuG+QMA0GCSqGSIb3DQEBCwUA
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
EwJSMzAeFw0yMTA5MDIwODMzMDZaFw0yMTEyMDEwODMzMDVaMBkxFzAVBgNVBAMT
Dnd3dy5tYXRyaXgub3JnMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEA
xVkNlBjdnI7aTcJFoMLSNoDCzKOx4DZRhLp92TvfiwtXBGP4XzzsExdsameeZcI6
eNEhCP7Cy8xTe9uTBxOmq06Uvf2XbJS2rltij7g+588a6ps7eaw7NJtMPRbpYrSh
sTYjQQXKl0E/btwacnlLft1DX0q2At+zqhhcRsd8+7iUpzTrU0arWkgKUW5gvmW6
GeHL9/BL9Yfyc5bH6K6ByvhiOC98wOcdyIxlWLM9XtPVcORQy0ms0HNSx6pN5caw
CtxcDIXcONucccJ+b2P7YXnplGc5qdM4yO9hOd3G5E+DYL7moJLyFs/wIYlpBw3S
4Kvs84Spl6Z4iQVGuWSt6ToQ4DBcSjMLMzenR1zelXiYfYKAYiBZOStETXOLtQNM
sRzw+YAStg13Xnk4o9oZXYKAHNfYsnFllj3eKkvT8otjNKhpDuByIsvwuL6l3KR3
87hXDboP5y3dThAQXtlnJVTradaB+Ffp9P2zRcG81v0hyNN3k0krygJV8ZIRUvCF
2dfuc+Wzl/aMIf0ZUohouk5zSid9Cvclb7e+v4I2dpkAky3MtnZp20IpKvk8857f
tzPEuYCR0mLhK12OzX8K5rGlNfcecfg/PQAlRr+cGWCkmTWIOzt3Tws/LPAN+y8k
Lsrj4M9Zq70l+edsWZUFLpT+KVCF3ap7R91bLzkIa5cCAwEAAaOCAw4wggMKMA4G
A1UdDwEB/wQEAwIFoDAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDAYD
VR0TAQH/BAIwADAdBgNVHQ4EFgQU60h02s4Bdg4lJtWSZe4fZzAsBRUwHwYDVR0j
BBgwFoAUFC6zF7dYVsuuUAlA5h+vnYsUwsYwVQYIKwYBBQUHAQEESTBHMCEGCCsG
AQUFBzABhhVodHRwOi8vcjMuby5sZW5jci5vcmcwIgYIKwYBBQUHMAKGFmh0dHA6
Ly9yMy5pLmxlbmNyLm9yZy8wgd0GA1UdEQSB1TCB0oINYWJvdXQucmlvdC5pbYIR
YXJld2VyZWFkeXlldC5jb22CDmxmcy5tYXRyaXgub3JnghhtYXRyaXgtY2xpZW50
Lm1hdHJpeC5vcmeCHG1hdHJpeC1mZWRlcmF0aW9uLm1hdHJpeC5vcmeCCm1hdHJp
eC5vcmeCCW1hdHJpeC50b4IKbW9kdWxhci5pbYIHcmlvdC5pbYIPc3BlYy5tYXRy
aXgub3Jnggl2ZWN0b3IuaW2CDnd3dy5tYXRyaXgub3Jngg53d3cubW9kdWxhci5p
bTBMBgNVHSAERTBDMAgGBmeBDAECATA3BgsrBgEEAYLfEwEBATAoMCYGCCsGAQUF
BwIBFhpodHRwOi8vY3BzLmxldHNlbmNyeXB0Lm9yZzCCAQQGCisGAQQB1nkCBAIE
gfUEgfIA8AB2APZclC/RdzAiFFQYCDCUVo7jTRMZM7/fDC8gC8xO8WTjAAABe6Xa
0lEAAAQDAEcwRQIhAO1PYxrVVhp/rKvu27WHONo0nd/AXt1z9vsQ0mg33OqsAiAm
mzM7zaTjFk7sNCYQCokhnraKXd98C6yZug2r2NMZWwB2AG9Tdqwx8DEZ2JkApFEV
/3cVHBHZAsEAKQaNsgiaN9kTAAABe6Xa0ocAAAQDAEcwRQIhAN0mlh7rsSh/7gya
5kCv2iKoVKgdCtd3rQNet47IHivLAiAkVKmqNEh4ZR9gE4tpd9WTUK4wU5gR2lWW
79cChdm7cjANBgkqhkiG9w0BAQsFAAOCAQEAZ+cRZSi1FeHF4meCZmV4pS2iDFaB
zMRns9FnS0zDCS29QrhSYEaJtdnANDi3E5FJqckF+U8Nafohx1BbIQ0tnjFk0zdX
A9L+7SjsgOLUxrAbXT9C0uWvEGbBZ8p1mJMtboKmgV0dLPYsNrGQ/2bK4SYxnyzy
bIyzNGfQEw3HtKRFybrcpHSjLvVMZL2QhhzZ7tpjUpREw++mRV3oBXVwU/SjG38E
9U7RlFQNTzLK2paZB8wEqWMMo6xt0Ot45oaJrGiIwUSM9ql3+19h1DP8RaQYpVV8
JDMmXios7EANq7pgvwPQ8qqCNRSB8+pqGbGfBNjmBc1bf7SlaTfUIjQ/Eg==
-----END CERTIFICATE-----
subject=CN = www.matrix.org

issuer=C = US, O = Let's Encrypt, CN = R3

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 5217 bytes and written 394 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 4096 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
DONE

While I was able to upgrade my server and openssl library, my concern is that for those that are running a homeserver on the latest version of macOS, they won't be able to federate

@hasanihunter
Copy link

hasanihunter commented Oct 6, 2021 •
edited

@callahad from the output from both machines in my reply above, it would seem that there are at least two different certificates being presented to a client depending on what version of TLS that the client supports. If that is the case, wouldn't a solution be to update the TLS 1.2 cert to one that is signed by the ISRG Root X1 root?

Assuming ISRG Root X1 is in their trust store of course.

@vmario89
Copy link
Author

vmario89 commented Oct 7, 2021

finally the fix was to run
openssl s_client -connect matrix.org:443 -showcerts -verify_return_error

that returned "verify error:num=2:unable to get issuer certificate"

i removed an old cert file from /etc/ssl/certs and it immediately started to work again

@richvdh
Copy link
Member

richvdh commented Oct 7, 2021

finally the fix was to run
openssl s_client -connect matrix.org:443 -showcerts -verify_return_error

this won't have fixed anything: it's just a diagnostic.

@vmario89
Copy link
Author

vmario89 commented Oct 7, 2021 •
edited

it helped me to fix, because it pointed out that the error lies in ssl, not in matrix. i checked the certs on ubuntu server and found out that one of the cert files was obsolete. after removing the statement , openssl s_client -connect matrix.org:443 -showcerts -verify_return_error did not return any error. The federation started working again because some buffered external server messages from previous days arrived me immediateley

@davehayes
Copy link

I know it's not synapse's responsibility to fix SSL issues, but a message indicating -why- a certificate verify failed would be extremely helpful to us in the trenches, and would probably lower the amount of github issues we (mistakenly) file. Some of us are not on the matrix "support" channel.

Thank you for at least considering this idea.

@richvdh
Copy link
Member

richvdh commented Oct 7, 2021

Unfortunately I don't think we get any more information back from the SSL libraries than "validation failed", so it's hard to do much about without rewriting openssl, which sounds... unattractive.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@callahad @hasanihunter @richvdh @babolivier @vmario89 @davehayes