Connection refused when setting port to zero and inside a docker container #276

KGHVerilator · 2021-06-22T19:00:01Z

I am trying to develop a test email server that should run in a docker container. The docker container is run with --network=host -P; though for unit tests I would like to use --network=none.

I am deliberately specifying that port 0 should be used (let the OS pick an available port). I then have a function to get the port that was actually used. The server code is

#! /usr/bin/env python3

import asyncio #pylint: disable=unused-import
from aiosmtpd.controller import Controller

class DemoEmailHandler:
    async def handle_DATA(self, server, session, envelope):
        message = {
            'peer' : session.peer,
            'mailfrom' : envelope.mail_from,
            'to': envelope.rcpt_tos,
            'data' : envelope.content}
        print(message)
        return '250 Message accepted for delivery'


class DemoEmailServer:
    def __init__(self):
        self._handler = DemoEmailHandler()
        self._server = Controller(
            handler=self._handler,
            hostname="0.0.0.0",
            port=0
        )
        self._server.start()
        self._port = self._server.server.sockets[0].getsockname()[1]

    def stop(self):
        """Tell the server to stop"""
        self._server.stop()

    def getPort(self) -> int:
        """Return the port the SMTP (email) server is actually on"""
        return self._port

SERVER = DemoEmailServer()
print(SERVER.getPort())
SERVER.stop()

I keep getting connection refused

Traceback (most recent call last):
  File "./demo_email_server.py", line 36, in <module>
    SERVER = DemoEmailServer()
  File "./demo_email_server.py", line 25, in __init__
    self._server.start()
  File "/usr/local/lib/python3.8/dist-packages/aiosmtpd-1.4.2-py3.8.egg/aiosmtpd/controller.py", line 223, in start
    self._trigger_server()
  File "/usr/local/lib/python3.8/dist-packages/aiosmtpd-1.4.2-py3.8.egg/aiosmtpd/controller.py", line 313, in _trigger_server
    s = stk.enter_context(create_connection((hostname, self.port), 1.0))
  File "/usr/lib/python3.8/socket.py", line 808, in create_connection
    raise err
  File "/usr/lib/python3.8/socket.py", line 796, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

when running inside a docker container. If I run on the host then the server starts up fine.

(My host and container are both Ubuntu 20.04, python 3.8.5, the container has aiosmtpd 1.4.2 installed from source).

The text was updated successfully, but these errors were encountered:

pepoluan · 2021-10-19T06:19:33Z

Ah, that's a new use case for me (setting port to 0)

Currently the code for Controller tries to connect to the specified port to trigger the server to activate once, in order to ensure that SMTP initializes correctly. It is a simple mechanism and currently does not consider the special port=0 case.

I'll see if I can add more intelligence to _trigger_server()

KGHVerilator · 2021-10-19T16:51:01Z

I like your package and find it very useful, thank you for your work. I have some python programming experience, but am only mediocre at it. I can help some with testing changes. There is some documentation that is not clear to me with setting up an SSL and TLS email server, if that is updated I would love to try it and say how it goes; assuming you are interested in that stuff. If this works in my docker containers, I can test stuff like that without worrying about my development environment. It has been a while since I looked at this. I may repeat a bit of what was in the log item. I will also try to give you my full use case, so you get my idea (good or bad) of how I think your software can be used. Up to you what you do with it. I am trying to test embedded devices that can send emails. They connect to an email server to store and forward the emails. I have a shared computer used in the tests. The embedded device is given the IP address and port of the email server on the shared computer. 1. Since lots of developers can be using the shared computer (including jenkins), I need the developer to startup, run, and shutdown the email server using a random port that is not privileged. 2. My test system needs to be contained and not dependent on the host system. Some of the products develop for a while then go dormant, only a few years later to be resurrected, hence containers with the test environment (and build environment). 3. My device needs to do three different kinds of email: - plain TCP/IP socket with everything in the clear (yuck but I am stuck with it) - SSL server email -TLS server for the email. I have been able to get TCP going, but the SSL and TLS servers are not working yet. 4. I have also tried setting up a subsystem test of my email server. The container running the email server also runs an email client with network=none (127.0.0.1 works). This allows us to have tests of the common code (email server) that is not dependent on any device. Again I got unencrypted working but not encrypted. 5. Our emails or both traditional text only emails, and HTML emails. I have not tried extracting the pictures we attach to our HTML emails yet, only working with text only ones. Kendrick

…

On 2021-10-19 12:19 a.m., Pandu E POLUAN wrote: Ah, that's a new use case for me (setting port to 0) Currently the code for |Controller| tries to connect to the specified port <https://urldefense.com/v3/__https://github.com/aio-libs/aiosmtpd/blob/master/aiosmtpd/controller.py*L416-L429__;Iw!!IOGos0k!31txFYuY9Rc4NFdQOkPOI0ejKqDtP3Cs3HXdjS2z64ULuOWGLtzGdODCN72pTf5W_suHBw$> to trigger the server to activate once. It is a simple mechanism and does not consider the special |port=0| case. I'll see if I can add more intelligence to |_trigger_server()| — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <https://urldefense.com/v3/__https://github.com/aio-libs/aiosmtpd/issues/276*issuecomment-946401243__;Iw!!IOGos0k!31txFYuY9Rc4NFdQOkPOI0ejKqDtP3Cs3HXdjS2z64ULuOWGLtzGdODCN72pTf5akcxw-A$>, or unsubscribe <https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AR3ZTVLW3XVFPVDQACMPVODUHUEX7ANCNFSM47EI37RQ__;!!IOGos0k!31txFYuY9Rc4NFdQOkPOI0ejKqDtP3Cs3HXdjS2z64ULuOWGLtzGdODCN72pTf5yHTvXEw$>. Triage notifications on the go with GitHub Mobile for iOS <https://urldefense.com/v3/__https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675__;!!IOGos0k!31txFYuY9Rc4NFdQOkPOI0ejKqDtP3Cs3HXdjS2z64ULuOWGLtzGdODCN72pTf5Hv8A5Kg$> or Android <https://urldefense.com/v3/__https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign*3Dnotification-email*26utm_medium*3Demail*26utm_source*3Dgithub__;JSUlJSU!!IOGos0k!31txFYuY9Rc4NFdQOkPOI0ejKqDtP3Cs3HXdjS2z64ULuOWGLtzGdODCN72pTf7Bp4KCww$>.

waynew · 2021-10-20T14:45:37Z

FWIW you can check out how I create an ssl server https://github.com/waynew/orouboros/blob/89905978a423e130e9ea2c57690b2e4a6cc48f3d/orouboros.py#L313

diazona · 2022-04-06T07:22:23Z

I've also run into this problem while trying to prepare pytest-dev/pytest-localserver for the removal of smtpd. I wanted to bump this issue to bring it some attention.

I'll probably wind up regretting saying this 😛 but it seems like a simple fix, just replace create_connection((hostname, self.port), 1.0) with create_connection(self.server_coro.sockets[0].getsockname(), 1.0), or something like that. Would you be open to accepting a pull request for that change @pepoluan? I'd be happy to prepare one, or at least put in a chunk of time to try to figure out a solution.

diazona · 2022-04-06T07:35:22Z

...and almost immediately I figured out why it's not that simple, with the mixing of synchronous and asynchronous code 😆 I knew I shouldn't have said anything. But I would still be quite willing to put some time into this, if it'd help.

waynew · 2022-06-09T05:15:53Z

Okay, so.... I know it's been a very long time since I've touched this, but - yeah, the fix is pretty straightforward, I think.

First off, yeah I was able to reproduce this.

Second off, I think the correct thing for us to do here is in the _create_server function where we do self.loop.create_server we could change it to:

server = self.loop.create_server(...)
if not self.port:
    self.port = server.sockets[0].getsockname()[1]
return server

By making that change locally, it all worked 👍 We'll want some test cases for that obviously, but it shouldn't be too hard to do!

waynew · 2022-06-09T06:53:47Z

Well, heck! The fix was pretty easy, though I did have to move the change to _trigger_server.

But I'm struggling with the test cases -- I think we actually have some tests that are failing on some more modern Pythons anyway 🤔

KGHVerilator · 2022-06-09T18:06:03Z

Hopefully this will work out; thank you for look into it.

alext · 2023-07-10T11:12:59Z

@waynew I was also hitting this, and I found the fix you describe worked for me as well, so I've raised it as #381.

pepoluan self-assigned this Oct 19, 2021

pepoluan added enhancement integration labels Oct 19, 2021

pepoluan added this to the 1.5 milestone Oct 19, 2021

alext linked a pull request Jul 10, 2023 that will close this issue

Allow setting port=0 to use OS-assigned port. #381

Open

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Connection refused when setting port to zero and inside a docker container #276

Connection refused when setting port to zero and inside a docker container #276

KGHVerilator commented Jun 22, 2021 •

edited by pepoluan

pepoluan commented Oct 19, 2021 •

edited

KGHVerilator commented Oct 19, 2021 via email

waynew commented Oct 20, 2021

diazona commented Apr 6, 2022 •

edited

diazona commented Apr 6, 2022

waynew commented Jun 9, 2022

waynew commented Jun 9, 2022

KGHVerilator commented Jun 9, 2022

alext commented Jul 10, 2023

Connection refused when setting port to zero and inside a docker container #276

Connection refused when setting port to zero and inside a docker container #276

Comments

KGHVerilator commented Jun 22, 2021 • edited by pepoluan

pepoluan commented Oct 19, 2021 • edited

KGHVerilator commented Oct 19, 2021 via email

waynew commented Oct 20, 2021

diazona commented Apr 6, 2022 • edited

diazona commented Apr 6, 2022

waynew commented Jun 9, 2022

waynew commented Jun 9, 2022

KGHVerilator commented Jun 9, 2022

alext commented Jul 10, 2023

KGHVerilator commented Jun 22, 2021 •

edited by pepoluan

pepoluan commented Oct 19, 2021 •

edited

diazona commented Apr 6, 2022 •

edited