Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Email Address Internationalization #51

Open
schmonz opened this issue Jul 30, 2019 · 9 comments · May be fixed by #197
Open

Email Address Internationalization #51

schmonz opened this issue Jul 30, 2019 · 9 comments · May be fixed by #197
Labels
enhancement New feature or request
Milestone

Comments

@schmonz
Copy link
Member

schmonz commented Jul 30, 2019

Russ Nelson has made very few updates to qmail.org (as of this writing, site's been down for a few weeks) in the last several years. One was pretty significant, though: Russ added Arnt Gulbrandsen's Email Address Internationalization (or EAI, or SMTPUTF8) patch to the very small "Recommended patches" section.

Coming from Russ, this is a strong recommendation. Should we include EAI in 1.08? I played with this patch a year or two ago, have integrated it into pkgsrc qmail, and would suggest from foggy memory that this is not an easy "yes":

  • I found and fixed at least one bug that prevented other qmail-smtpds from accepting mail from an EAI-enabled qmail-remote
  • IIRC, the patch claims it applies atop TLS but actually the SMTP AUTH patch is what it expects to be present
  • I don't remember the code being as clear and expressive as I'd wish for (possibly related to why it had at least one bug)

For reference:

@DerDakon
Copy link
Member

I remember that I once looke into this for implementing in Qsmtp. IIRC it was not that easy, so I would vote this for 1.09 or so, we have clearer things to do in 1.08.

@schmonz schmonz added the enhancement New feature or request label Jul 30, 2019
@schmonz
Copy link
Member Author

schmonz commented Aug 2, 2019

For reference, here's a copy of my analysis from the qmail list:

I haven't noticed any problems with qmail-smtpd, but I think the patch adds a bug to qmail-remote.

To reproduce the bug:

  1. Install a vanilla netqmail into /var/qmail-there

  2. Apply the EAI patch, then install into /var/qmail-here

  3. In /var/qmail-here/control/smtproutes, set qmail-there as qmail-here's smarthost

  4. Run /var/qmail-there/bin/qmail-smtpd, wrapped in recordio

  5. Run /var/qmail-here/bin/qmail-inject, sending to postmaster@qmail-there

  6. Note that the message stays in qmail-here's queue

  7. Inspect qmail-there's qmail-smtpd log

  8. Observe that SMTP commands and message body lines are CRLF-terminated, but header lines are bare-LF-terminated

Temporarily wrapping qmail-there's qmail-smtpd with fixcrio then allows the message through.

The problem does not depend on whether the message is ASCII-only or UTF-8.

My analysis: qmail-remote.c:blast() normally munges every LF into CRLF. But when the EAI patch adds a "firstpart", those line endings don't get munged in the same way. The one-liner below munges firstpart's line endings a bit earlier. It appears to fix my bug, and maybe doesn't introduce new ones.

--- qmail-remote.c.orig 2017-07-30 18:24:37.000000000 +0000
+++ qmail-remote.c
@@ -253,6 +253,7 @@ void checkutf8message()
     if (r == 0) break;
     if (r == -1) temp_read();

+    if (ch == '\n' && !stralloc_append(&firstpart,"\r")) temp_nomem();
     if (!stralloc_append(&firstpart,&ch)) temp_nomem();

     if (ch == '\r')

@schmonz schmonz added this to the 1.90 milestone Nov 20, 2020
@mbhangui
Copy link
Contributor

mbhangui commented Dec 3, 2020

The patch has one more tiny problem. The variable asciihost should be null terminated after stralloc_copys(&asciihost, ascii)

@@ -702,9 +793,17 @@
       relayhost[i] = 0;
     }
     if (!stralloc_copys(&host,relayhost)) temp_nomem();
+  } else {
+    char * ascii = 0;
+    host.s[host.len] = '\0';
+    switch (idn2_lookup_u8(host.s, (uint8_t**)&ascii, IDN2_NFC_INPUT)) {
+      case IDN2_OK: break;
+      case IDN2_MALLOC: temp_nomem();
+      default: perm_dns();
+    }
+    if (!stralloc_copys(&asciihost, ascii)) temp_nomem();
   }
 
-
   addrmangle(&sender,argv[2],&flagalias,0);
  
   if (!saa_readyplus(&reciplist,0)) temp_nomem();
@@ -723,7 +822,7 @@
 
  
   random = now() + (getpid() << 16);
-  switch (relayhost ? dns_ip(&ip,&host) : dns_mxip(&ip,&host,random)) {
+  switch (relayhost ? dns_ip(&ip,&host) : dns_mxip(&ip,&asciihost,random)) {

@mbhangui
Copy link
Contributor

mbhangui commented Dec 3, 2020

This is EH's version. https://www.fehcom.de/sqmail/doxygen/qmail-remote_8c_source.html

@mbhangui
Copy link
Contributor

mbhangui commented Dec 4, 2020

I ended up testing both versions.

  1. Arnt's version has two bugs. One which you mentioned above and another one regarding null termination
  2. EH's version has a smaller parser but fails if the Received header continuation line starts with tabs or starts with more than two white spaces. I have fixed EH's version by skipping extra white spaces.
int utf8read()
{
  int r;
  int i;
  int received = 0;
  char ch;

  /* we consider only our own last written header */

  for (;;) {
    r = substdio_get(subfdin,&ch,1);
    if (r == 0) break;
    if (r == -1) temp_read();

    if (ch == '\n') {
     if (!stralloc_append(&header,"\r")) temp_nomem(); /* received.c does not add '\r' */
     if (!stralloc_append(&header,"\n")) temp_nomem();  
     if (case_startb(receivedline.s,5,"Date:")) return 0;         /* header to quit asap */
     if (case_startb(receivedline.s,14,"Received: from")) received++; /* found Received header */
     if (received) {
      for (i = 0;i < receivedline.len; i++)
        if (*(receivedline.s+i) != ' ' && *(receivedline.s+i) != '\t')
         break;
      if (case_startb(receivedline.s + i,3,"by ")) {
        for (i += 3; i < receivedline.len; ++i)  
         if (*(receivedline.s+i) == ' ' || *(receivedline.s+i) == '\t')
          if (case_startb(receivedline.s+i+1,9,"with UTF8")) return 1;
        return 0;
      }
     }
     if (!stralloc_copys(&receivedline,"")) temp_nomem();
    } else {
     if (!stralloc_append(&header,&ch)) temp_nomem();
     if (!stralloc_catb(&receivedline,&ch,1)) temp_nomem();
    }
  }
  return 0;
}

@schmonz
Copy link
Member Author

schmonz commented Dec 4, 2020

Awesome! At a glance, that function looks like it might lend itself to being unit tested. Given the short but already buggy history of SMTPUTF8 in qmail, would you be willing to write some tests for common and edge cases?

@DerDakon
Copy link
Member

DerDakon commented Dec 4, 2020

That function is also a bit inefficient at a glance. For me it looks like if it is called multiple times, so I would say it should store it's result in a static variable and return early if that already has a value.

@mbhangui
Copy link
Contributor

mbhangui commented Dec 4, 2020

The function is called just once, before the blast() function, and exits the moment it sees the UTF8 in the received header or the Date: header, whichever comes first.

@DerDakon
Copy link
Member

DerDakon commented Dec 4, 2020

You are right, it's utf8received() that has this problem.

@schmonz schmonz linked a pull request Dec 16, 2020 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants