MailNotifier fails to send email #5943

muks · 2021-03-23T21:55:32Z

I have a MailNotifier configured like this:

s = reporters.MailNotifier(
    fromaddr='buildbot@example.org',
    extraRecipients=['notify@example.org'],
    sendToInterestedUsers=False,
    relayhost='mail.example.org',
    smtpPort=587,
    useTls=True,
    smtpUser='buildbot@example.org',
    smtpPassword='randomstring')
c['services'].append(s)

Notification mails are not sent. The following messages are logged in twistd.log:

2021-03-23 21:47:39+0000 [-] Starting factory <twisted.mail.smtp.ESMTPSenderFactory object at 0x7f7742e74d90>
2021-03-23 21:47:39+0000 [ESMTPSender,client] Got exception when handling reporter events
        Traceback (most recent call last):
          File "/home/buildbot/venv/lib64/python3.8/site-packages/twisted/internet/defer.py", line 1514, in gotResult
            current_context.run(_inlineCallbacks, r, g, status)
          File "/home/buildbot/venv/lib64/python3.8/site-packages/twisted/internet/defer.py", line 1443, in _inlineCallbacks
            result = current_context.run(result.throwExceptionIntoGenerator, g)
          File "/home/buildbot/venv/lib64/python3.8/site-packages/twisted/python/failure.py", line 500, in throwExceptionIntoGenerator
            return g.throw(self.type, self.value, self.tb)
          File "/home/buildbot/venv/lib64/python3.8/site-packages/buildbot/reporters/base.py", line 95, in _got_event
            log.err(e, 'Got exception when handling reporter events')
        --- <exception caught here> ---
          File "/home/buildbot/venv/lib64/python3.8/site-packages/buildbot/reporters/base.py", line 93, in _got_event
            yield self.sendMessage(reports)
        twisted.mail._except.TLSRequiredError: 502 Server does not support secure communication via TLS / SSL
        <<< 250-8BITMIME
        <<< 250-DSN
        <<< 250-SMTPUTF8
        <<< 250 CHUNKING

This used to work before, so it appears something has changed in the software versions. pyOpenSSL is available in the venv.

(venv) [buildbot@dev ~]$ pip list | grep -i -E '(pyopenssl|cryptography)'
cryptography            3.4.6
pyOpenSSL               20.0.1
(venv) [buildbot@dev ~]$

The server does support STARTTLS on port 587. We all use it with various email clients for email submission. As an example, Gerrit is able to send emails via this same email submission service.

Edit: The email server is Postfix:

[muks@mx ~]$ rpm -q postfix
postfix-3.5.9-1.fc32.x86_64
[muks@mx ~]$ rpm -q openssl
openssl-1.1.1i-1.fc32.x86_64
[muks@mx ~]$

The text was updated successfully, but these errors were encountered:

muks · 2021-03-23T22:05:35Z

Sample session with openssl s_client to prove that STARTTLS is available on port 587:

[buildbot@dev ~]$ openssl s_client -connect mail.example.org:587 -starttls smtp
CONNECTED(00000003)
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = [REDACTED]
verify return:1
---
Certificate chain
 0 s:CN = [REDACTED]
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:O = Digital Signature Trust Co., CN = DST Root CA X3
---
Server certificate
-----BEGIN CERTIFICATE-----

... [REDACTED] ...

-----END CERTIFICATE-----
subject=CN = REDACTED]

issuer=C = US, O = Let's Encrypt, CN = R3

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: ECDSA
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 2871 bytes and written 429 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 256 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
250 CHUNKING
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
    Session-ID: [REDACTED]
    Session-ID-ctx:
    Resumption PSK: [REDACTED]
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 7200 (seconds)
    TLS session ticket:
	... [REDACTED] ...

    Start Time: 1616536878
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
quit
221 2.0.0 Bye
closed
[buildbot@dev ~]$

muks · 2021-03-27T12:53:22Z

Briefly reading through the twisted/src/twisted/mail/smtp.py code, the output logged (after EHLO) in the report above by SMTPClient.lineReceived() appears incomplete compared to what the server returns. The output logged above is:

        twisted.mail._except.TLSRequiredError: 502 Server does not support secure communication via TLS / SSL
        <<< 250-8BITMIME
        <<< 250-DSN
        <<< 250-SMTPUTF8
        <<< 250 CHUNKING

whereas the SMTP server on port 587 returns the following messages after EHLO is sent by the client:

250-mail.example.org
250-PIPELINING
250-SIZE 163840000
250-VRFY
250-ETRN
250-STARTTLS
250-ENHANCEDSTATUSCODES
250-8BITMIME
250-DSN
250-SMTPUTF8
250 CHUNKING

It appears the first few lines that include 250-STARTTLS are lost, and so ESMTPClient.esmtpState_serverConfig() doesn't include it in the items dict, and canTLS is False in ESMTPClient.tryTLS().

LuluDavid · 2021-09-09T14:59:04Z

Have exactly the same error here, but it works again if I downgrade Twisted to 20.3.0.
Seems that it could come from this issue on Twisted that led to the migration from 20.3.0 to 21.2.0.

hadleyrich · 2021-09-15T01:55:43Z

Have exactly the same error here, but it works again if I downgrade Twisted to 20.3.0.
Seems that it could come from this issue on Twisted that led to the migration from 20.3.0 to 21.2.0.

Ditto

Swansky · 2022-03-02T13:11:29Z

Hi, this problem is always present.
The downgrade from twisted to 20.3.0 was working for the office365 smtp server until microsoft removed the TLS 1.0, 1.1 support for TLS1.2. Today I am facing a critical deadlock where sending mail does not work anymore. I have tried all recent twisted packages (from 20.3.0) none of them fixes the problem. Disabling 'useTls' for 'useSmtps' does not change anything.
The error message is the same as in the first post, and so is my configuration.
If you have any idea, even temporary, I'm interested.
Thank you.

tenta4 · 2022-04-18T15:11:09Z

This is because new Twisted 21.2.0+ required hostname argument in the ESMTPSenderFactory
So in buildbot 2.10.X I fixed it by next patch

index 54e9c88..79d2886 100644
--- a/reporters/mail.py
+++ b/reporters/mail.py
@@ -351,7 +351,7 @@ class MailNotifier(ReporterBase):
             unicode2bytes(self.smtpUser), unicode2bytes(self.smtpPassword),
             parseaddr(self.fromaddr)[1], recipients, BytesIO(s),
             result, requireTransportSecurity=self.useTls,
-            requireAuthentication=useAuth)
+            requireAuthentication=useAuth, hostname=self.relayhost)
 
         if self.useSmtps:
             reactor.connectSSL(self.relayhost, self.smtpPort,

muks · 2022-04-22T06:10:06Z

@p12tic Please can you take a look at the patch in the previous comment, and merge it if it appears correct?

Edit: The patch above did not appear to make any difference in my case. The issue still remained.

doberkofler · 2022-05-24T16:27:51Z

This is because new Twisted 21.2.0+ required hostname argument in the ESMTPSenderFactory So in buildbot 2.10.X I fixed it by next patch

index 54e9c88..79d2886 100644
--- a/reporters/mail.py
+++ b/reporters/mail.py
@@ -351,7 +351,7 @@ class MailNotifier(ReporterBase):
             unicode2bytes(self.smtpUser), unicode2bytes(self.smtpPassword),
             parseaddr(self.fromaddr)[1], recipients, BytesIO(s),
             result, requireTransportSecurity=self.useTls,
-            requireAuthentication=useAuth)
+            requireAuthentication=useAuth, hostname=self.relayhost)
 
         if self.useSmtps:
             reactor.connectSSL(self.relayhost, self.smtpPort,

I have tried this workaround but was not successful.

cryptable · 2022-07-26T10:31:04Z

This workaround by adding the hostname to the ESMTPSenderFactory solved my problem sending mails through gmail. Can this be integrated or shall I ask for a pull request?

tenta4 · 2022-07-27T09:43:58Z

Created PR #6579

muks · 2022-07-27T09:49:46Z

Created PR #6579

Please don't resolve this ticket after merging PR #6579 as it does not fix the problem for the reporter of this ticket, and others on this ticket. What PR #6579 fixes may in fact be a different bug that has similar symptoms.

The third comment on this ticket describes what occurs for this bug, that PR #6579 is unlikely to solve (from its diff).

muks · 2022-07-27T10:08:52Z

Perhaps an msmtpd daemon, running on the same host as the Buildbot master, acting as a proxy to the real SMTP submission service, would offer a workaround for the bug.

https://marlam.de/msmtp/msmtp.html#Minimal-SMTP-server-_0028msmtpd_0029

cryptable · 2022-07-27T12:33:15Z

I use buildbot master on Linux (ubuntu 22.04) and use Ansible now to patch the ESMTPSenderFactory. It would be nice I don't have to patch that mail.py file when installing builldbot. Any advise would be welcome if alternative is possible.

tardyp · 2022-07-29T18:45:56Z

@muks could you please try upgrading twisted and apply #6579.
it appears to me that the original issue is actually a Twisted issue.

doberkofler · 2022-08-29T05:52:35Z

@tardyp Still not able to send email via gmail from buildbot master 3.6.0 running in docker container!

Python version: 3.9.2
Buildbot version: 3.6.0
Twisted version: 22.4.0

master-config-buildbot-1  | 2022-08-29 05:16:35+0000 [ESMTPSender,client] Got exception when handling reporter events
master-config-buildbot-1  |     Traceback (most recent call last):
master-config-buildbot-1  |       File "/buildbot_venv/lib/python3.9/site-packages/twisted/internet/defer.py", line 1750, in gotResult
master-config-buildbot-1  |         current_context.run(_inlineCallbacks, r, gen, status)
master-config-buildbot-1  |       File "/buildbot_venv/lib/python3.9/site-packages/twisted/internet/defer.py", line 1656, in _inlineCallbacks
master-config-buildbot-1  |         result = current_context.run(
master-config-buildbot-1  |       File "/buildbot_venv/lib/python3.9/site-packages/twisted/python/failure.py", line 514, in throwExceptionIntoGenerator
master-config-buildbot-1  |         return g.throw(self.type, self.value, self.tb)
master-config-buildbot-1  |       File "/buildbot_venv/lib/python3.9/site-packages/buildbot/reporters/base.py", line 113, in _got_event
master-config-buildbot-1  |         log.err(e, 'Got exception when handling reporter events')
master-config-buildbot-1  |     --- <exception caught here> ---
master-config-buildbot-1  |       File "/buildbot_venv/lib/python3.9/site-packages/buildbot/reporters/base.py", line 111, in _got_event
master-config-buildbot-1  |         yield self.sendMessage(reports)
master-config-buildbot-1  |     twisted.mail._except.TLSRequiredError: 502 Server does not support secure communication via TLS / SSL

c['services'] = []

from emailLookup import EmailLookup

generator = reporters.BuildStatusGenerator(
	mode=('change'),
	)

mn = reporters.MailNotifier(
	fromaddr=private.auth_username,
	sendToInterestedUsers=True,
	extraRecipients=[],
	useTls=True,
	relayhost='smtp.gmail.com',
	smtpPort=587,
	smtpUser=private.auth_username,
	smtpPassword=private.auth_password,
	lookup=EmailLookup(),
	generators=[generator],
	)

c['services'].append(mn)

p12tic added the bug label Mar 29, 2021

doberkofler mentioned this issue May 24, 2022

Error when trying to send email using gmail #6451

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MailNotifier fails to send email #5943

MailNotifier fails to send email #5943

muks commented Mar 23, 2021 •

edited

muks commented Mar 23, 2021

muks commented Mar 27, 2021 •

edited

LuluDavid commented Sep 9, 2021

hadleyrich commented Sep 15, 2021

Swansky commented Mar 2, 2022

tenta4 commented Apr 18, 2022 •

edited

muks commented Apr 22, 2022 •

edited

doberkofler commented May 24, 2022

cryptable commented Jul 26, 2022

tenta4 commented Jul 27, 2022

muks commented Jul 27, 2022

muks commented Jul 27, 2022 •

edited

cryptable commented Jul 27, 2022 •

edited

tardyp commented Jul 29, 2022

doberkofler commented Aug 29, 2022

MailNotifier fails to send email #5943

MailNotifier fails to send email #5943

Comments

muks commented Mar 23, 2021 • edited

muks commented Mar 23, 2021

muks commented Mar 27, 2021 • edited

LuluDavid commented Sep 9, 2021

hadleyrich commented Sep 15, 2021

Swansky commented Mar 2, 2022

tenta4 commented Apr 18, 2022 • edited

muks commented Apr 22, 2022 • edited

doberkofler commented May 24, 2022

cryptable commented Jul 26, 2022

tenta4 commented Jul 27, 2022

muks commented Jul 27, 2022

muks commented Jul 27, 2022 • edited

cryptable commented Jul 27, 2022 • edited

tardyp commented Jul 29, 2022

doberkofler commented Aug 29, 2022

muks commented Mar 23, 2021 •

edited

muks commented Mar 27, 2021 •

edited

tenta4 commented Apr 18, 2022 •

edited

muks commented Apr 22, 2022 •

edited

muks commented Jul 27, 2022 •

edited

cryptable commented Jul 27, 2022 •

edited