New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
WIP EAI (SMTPUTF8 for SMTP, UTF8=ACCEPT for IMAP) #190
base: main
Are you sure you want to change the base?
Conversation
This adds the capability and accepts UTF8 quoted-strings. The SEARCH command is changed as required by RFC 6855, section 3, final paragraph. There are no changes to the searching, as 6855 only changes changes the syntax. This does not add any kind of downgrading. Before this change, Dovecot would accept an APPEND of a message such as From: grå@grå.org Subject: grå ... and would just-send-8 that to any IMAP clients. This change maintains that policy.
This adds support for SMTPUTF8 (RFC 6531) to the LMTP and Submission services. The Submission service advertises this extension only if its upstream relay does, and takes some care to avoid a misunderstood error message.
Thanks, we'll take a look. |
Hi, one big issue with this code is that it mainly just makes dovecot accept UTF8, yet it does very little in the way of actually handling it. It also does not deal with unicode normalizations required for recipient handling. So in short, while we do really appreciate the effort, we would need this to actually take care of the utf-8 we are accepting, and not just wish for the best and store it, which seems to be missing from your TODO list. |
Thanks for your quick response. The patch doesn't do anything about the UTF8 because RFCs 589x, 6531 and 6855 demand nothing of a server such as Dovecot. To name three examples:
|
Actually, let's do it differently. Why don't you just make some imaptest tests that fail, and I'll make them pass. That'll explain concisely what you have in mind. Does that sound good to you? |
One more question. This PR's goal ist limited in scope to EAI support like gmail's: Users can receive mail from and send mail to grå@grå.org and deal with that mail as with all mail about grå, It does not aim to host a domain such as grå.org itself. Is that an acceptable scope to you? I could do a separate PR to support hosting if you want to, but I'd really prefer that to be a separate PR. |
The problem is mostly that if i send email to ℌdž@domain.com, it should go to hdž@domain.com. If there is no normalization done, these two are considered different user. This is handled by https://www.rfc-editor.org/rfc/rfc8265, which applies to sender/recipient names too. That said, this also needs to handle other headers as well. This is governed by https://www.rfc-editor.org/rfc/rfc6532. Before we start coming up with examples, why not take a moment to read these? |
I know both of them. I should ;) I believe Dovecot escapes the requirements in 8265 at present. (Being able to host grå.org would change this.) 8265 applies to software that accepts addresses from outside, does some sort of comparison and then does something differently based on the result of the comparison. Dovecot as it stands accepts addresses, but does nothing differently based on the result. For example, the submission relays all addresses to the backend server. 6532 changes a number of limits and lifts a number of restrictions that matter to this PR, but AFAICT Dovecot already lifted those long ago, so no changes were necessary. 6532 has implications for DSN generators. As far as I can tell, none of the code I touched will generate DSNs, but the Sieve code will. Are you saying that you'd require Sieve to be updated as well? |
Well we derive the username from the destination address, so at minimum that has to work, as well as We can consider doing this in increments, as long as we clearly refuse stuff we cannot handle correctly. |
Oh and sieve is these days pretty wedded to dovecot, so it should not break either. |
I could make a followup PR to support local users with UTF8 names. The goal of this PR is more restricted (the same level of support as gmail currently has — foo@gmail.com can send mail to foo@grå.org, but there cannot be a grå@gmail.com). It would be much easier for me to get management approval for more work if gmail-level support is merged. The followup would then add support for local users with EAI addresses and for Sieve. This PR is a WIP and I can put more W into it, but I can't put unlimited work into it without an assurance that the work will be merged, see? |
ok. I'll have to discuss this internally then. |
At minimum SEARCH FROM has to work. Even if we only accept utf8 senders. |
I absolutely agree that SEARCH FROM has to work. (It's a bit tricky, e.g. if the message contains a DKIM or PGP signature over the bytes as received.) How do you prefer automated tests in PRs such as this? |
we have internal ci with tests (provide some shell script / python script) and unit tests if possible. imaptest script is ok too, although not sure if it can test this, esp. the problem cases. |
Could you possibly link to an example in the style you most prefer? |
we'll have to adapt the ci test in any case. |
Sadly, RFC 8625 is required even in the gmail-level support I had in mind for this PR. Sieve tests such as |
Hi, I wrote automatic tests for this (and did a little more work on the PR too). I believe that the optimal way to support EAI in Dovecot is with three PRs, this is one of them and logically the first.
I can write the second and third, but I need this merged, or an assurance that it will be merged. |
Thanks. We'll take a look and let you know. |
@cmouse Hi. Any updates on your decisions? I suppose before the SMTPUTF8 support in
|
@HLFH Yes, you need to disable smtputf8 in postix. |
@arnt it seems to pass current tests so that's at least good. There were some boolean issues, but we can take care of those if we merge this. |
You don't really need to disable SMTPUTF8 in postfix — senders get the same error message in both cases. Disabling and enabling both have advantages, and both are really small advantages ;) To whom should I send the test? UID SEARCH for the same word normalised and denormalised gave the same result, so something or other already handles normalisation. |
This is based on code from someone called 'vk'. It's incomplete (needs RFC8265 support at least), but works for him/her.
This stores the unicode form of domains in all indexes, meaning that searching uses and serverside parsing shows the human-readable form of all addresses.
I updated the PR so that adding a message with |
Hi, we'll take a look. |
Moved as internal merge request. |
Hi,
this is a WIP patch to support EAI. Some remarks:
I'd appreciate comments.