Communication via STDIN/STDOUT

[chamilton-ccn]

Hello,

I'm prototyping a content filter that is intended to be initiated via Postfix's spawn service. Per the documentation, this behaves in similar fashion to services initiated via inetd/xinetd. Basically STDIN/STDOUT are "connected" to the network socket via spawn during initialization. aiosmtpd doesn't appear to have any native way of utilizing STDIN/STDOUT. The closest thing I've been able to come up with is communicating via an intermediary UNIX domain socket. The filter, example input, and postfix config are all included below.

Basically what I'm asking is this: In its current form, is there any way to use STDIN/STDOUT with aiosmtpd and if so, how might I go about doing that?

---

filter.py

#!/usr/bin/env python -u
import sys
import os
import fcntl
import syslog
import socket
from email import message_from_bytes
from email.policy import default
from aiosmtpd.smtp import SMTP
from aiosmtpd.controller import UnixSocketController


syslog.openlog("filter-test", 0, syslog.LOG_MAIL)
socket_path = '/tmp/smtp_socket~'

class ExampleSMTPHandler:
    async def handle_RCPT(self, server, session, envelope, address, rcpt_options):
        envelope.rcpt_tos.append(address)
        return '250 OK' 

    async def handle_DATA(self, server, session, envelope):
        message = message_from_bytes(envelope.content, policy=default)
        syslog.syslog(str(type(message)))
        syslog.syslog(str(message.get_body()))
        for attachment in message.iter_attachments():
            syslog.syslog(str(attachment))
        return '250 Message accepted for delivery'

controller = UnixSocketController(ExampleSMTPHandler(), unix_socket=socket_path)
controller.start()
s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
s.connect(socket_path)

# Set socket to nonblocking
s.setblocking(False)

# Set stdin nonblocking
orig_fl = fcntl.fcntl(sys.stdin, fcntl.F_GETFL)
fcntl.fcntl(sys.stdin, fcntl.F_SETFL, orig_fl | os.O_NONBLOCK)

def socket_readline_to_stdout(my_socket) -> str:
    line = ''
    try:
        while True:
            part = my_socket.recv(1).decode('utf-8')
            line += part
            if line.endswith('\r\n'):
                break
    except socket.error as e:
        pass
    sys.stdout.write(line)
    return line

def stdin_readline_to_socket(my_socket) -> str:
    line = ''
    while True:
        part = sys.stdin.read(1)
        line += part
        if part == '': 
            break
    try:
        my_socket.send(line.encode('utf-8'))
    except socket.error as e:
        pass
    return line

while True:
    l = socket_readline_to_stdout(s)
    if l == '':
        l = stdin_readline_to_socket(s)
    if l == 'QUIT\r\n':
        sys.exit(0)

The code above is an extremely basic example of how this can be accomplished, but it's ugly, clunky and likely error prone with all but the most basic input. Its only purpose is to demonstrate the use of an intermediary socket.

example input

ehlo test
mail from:<some.user@domain.tld>
rcpt to:<some.user2@domain2.tld>
data
From: Some User <some.user@domain.tld>
To: Some User <some.user2@domain2.tld>
Subject: This is a test
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="X-X-X-X-X-X-X-X-text boundary"

--X-X-X-X-X-X-X-X-text boundary
Content-Type: text/plain

This is a test!

--X-X-X-X-X-X-X-X-text boundary
Content-Type: application/pdf; name="file.pdf"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="file.pdf"

[ ... base64 content here ... ]

--X-X-X-X-X-X-X-X-text boundary

.

---

master.cf

#### Configure the SMTP listener on a custom port
127.0.0.1:2587  inet    n       -       n       -       -       smtpd
        -o syslog_name=postfix/content-filter-test
        -o smtpd_tls_security_level=may
        -o content_filter=filter-test:[127.0.0.1]:10025
        -o receive_override_options=no_address_mappings

##### Configure the smtp client named in the content_filter argument above
filter-test unix    -       -       n       -       10      smtp
        -o smtp_send_xforward_command=yes
        -o disable_mime_output_conversion=yes
        -o smtp_generic_maps=
        -o syslog_name=postfix/content-filter-test

##### Configure the content filter (receive)
127.0.0.1:10025 inet    n       n       n       -       10      spawn
        -o syslog_name=postfix/content-filter-test
        user=filter-user    argv=/path/to/content/filter.py

#### Configure the content filter (reinjection path)
#127.0.0.1:10026 inet    n       -       n       -       10      smtpd
#        -o content_filter= 
#        -o receive_override_options=no_unknown_recipient_checks,no_header_body_checks,no_milters
#        -o smtpd_helo_restrictions=
#        -o smtpd_client_restrictions=
#        -o smtpd_sender_restrictions=
##        # Postfix 2.10 and later: specify empty smtpd_relay_restrictions.
#        -o smtpd_relay_restrictions=
#        -o smtpd_recipient_restrictions=permit_mynetworks,reject
#        -o mynetworks=127.0.0.0/8
#        -o smtpd_authorized_xforward_hosts=127.0.0.0/8
#        -o syslog_name=postfix/content-filter-test/REINJECT

Note: I've disabled the reinjection path since the filter in the first example doesn't actually pass the message content back to the server; I didn't have a need for this during testing but at some point the filter would handle passing the message back. The reinjection path configuration is included above, for sake of completeness.

[pepoluan]

The biggest challenge is that aiosmtpd directly couples to asyncio's Transport feature.

In there, there are a HUGE number of hairy things asyncio has to do to make TCP streams async.

So I don't think aiosmtpd can easily support native STDIN/STDOUT.

[chamilton-ccn]

Certainly! Though it bears repeating the above is only a proof-of-concept. If someone wanted to use this method (intermediary UNIX domain socket) for production code, they'd need to put more work into it.

[waynew]

@chamilton-ccn would you be interested in writing up a bit of how-to? If you don't feel the instructions are suffuciently polished to include in the docs, just add it in the discussions, that way whoever picks it up in the future will have a jumping off point 👍

[chamilton-ccn]

@waynew Sure but it might be a while before I can get around to it, upcoming holidays and current workload and all that. As it stands, I think the example above is an ok starting point. My main issues with it are that I haven't tested it beyond the sample input I've provided, and no matter what, the handle_RCPT and handle_DATA methods return successful statuses (e.g., there's no input validation or error handling whatsoever). If someone wanted to use this in production, they'd have to deal with those cases. Stating all of this in case someone gets around to it before I'm able to :-)

[pepoluan]

@chamilton-ccn no worries 😄

For the time being we can link to this discussion, with some emphasis on "Proof of Concept" and "example" or words to that effect, plus maybe additional emphasis that one needs to adopt/adapt the strategy shown in accordance to their own needs.

"Treat this like a Stack Overflow answer: Do NOT copy-paste! Study it, adapt, and infuse in your own code according to your own flow"

[chamilton-ccn]

@pepoluan

"Treat this like a Stack Overflow answer: Do NOT copy-paste! Study it, adapt, and infuse in your own code according to your own flow"

I love it 😆