Messages moved to DeltaChat folder on Mailcow instance received in wrong order #5339
Comments
|
I have tested with nine.testrun.org which also runs Dovecot, there moving the messages did not change Unselecting and reselecting the folder to clear |
|
I have looked at my INBOX on testrun.org (I do not move messages to DeltaChat folder) and there seems to be some problem in INTERNALDATE timestamping on that day, see UID 181201 and UID 181202 having weird timestamps) But the messages mentioned in the first post have these UID+dates in my case: Tested the whole INBOX with this script, there are more cases of internaldate problems: #!/usr/bin/env python3
import imaplib
from imaplib import IMAP4_SSL
import sys
login = sys.argv[1]
password = sys.argv[2]
with IMAP4_SSL("testrun.org") as M:
M.login(login, password)
M.select("INBOX", readonly=True)
status, resp = M.uid("FETCH", "1:*", "(INTERNALDATE)")
prev_date = None
for line in resp:
timestruct = imaplib.Internaldate2tuple(line)
if prev_date and prev_date > timestruct:
print(line.decode("ascii"))
prev_date = timestructRunning the script on my testrun account I got this output: System seem to have been rebooted on Mar 5 and not Mar 6, so reboots do not explain clock jumps. On my nine.testrun.org account there are zero problems of this kind. |
|
Relying on Or we can avoid ordering by |
|
We can also do nothing because it does not happen often and the bug is actually on the server side, it should ensure that the clock does not jump. |
I think first of all the reason of clock jumps should be found and anyway Mailcow could guarantee that INTERNALDATE is at least monotonically increasing. But there's still a question what to do if two messages have the same. |
We sort by UID in this case, there is no problem with UIDs. Even sorting only by UIDs work, it only does not work for Gmail. I looked on the messages on testrun.org from the server: Here Also messages with Mail with UID=181201 was apparently indeed received at 10:42:04 +0100, but only saved to the mailbox at 17:12. Could be it somehow got stuck in LMTP delivery e.g. because of rspamd? I considered using |
Sidenote:
|
…bracket error This is a test reproducing the problem in <#5339>. Fix would be to avoid reordering on the server side, so the test checks that the unverified message is replaced with a square bracket error as expected if messages arrive in the wrong order.
…bracket error This is a test reproducing the problem in <#5339>. Fix would be to avoid reordering on the server side, so the test checks that the unverified message is replaced with a square bracket error as expected if messages arrive in the wrong order.
But it looks like using INTERNALDATE (mtime) is not so bad. What can be the reasons of its change after a mail is received? And anyway using mtime is a server implementation detail. The problem is that messages are delayed arbitrarily, so they are anyway received in the wrong order if a client is constantly online and has no chance to order them properly. Maybe if we see an out-of-order (delayed) message (this needs some heuristics however, maybe using |
In this particular case there was no such problem, messages were received in the correct order and have correct UID order. Constantly online client would have downloaded them in the right order.
In this case editor message replaced with square bracket error would need reprocessing after receiving "late" Member added message. Processing old messages that previously failed validation is adding too much complexity and will create more bugs e.g. if such messages are themselves changing group membership. Currently Delta Chat always processes messages no more than once and I would rather not change it without Topological sort based on |
…bracket error This is a test reproducing the problem in <#5339>. Fix would be to avoid reordering on the server side, so the test checks that the unverified message is replaced with a square bracket error as expected if messages arrive in the wrong order.
+1, no message reprocessing of course. Looks like there's nothing special to be done for out-of-order (delayed) messages. For messages downloaded later we only replace placeholder messages in the db.
If the subject problem will be fixed for Mailcow, it seems the topological sort can be helpful only for Gmail. It can also help in case of message delivery delays, but only if a client is not constantly online, but luckily fetches messages at the proper moments. |
We recently had a problem with two messages received in wrong order: "member added" message and following editor.xdc message.
Two messages received by c@testrun.org
running Delta Chat Android version 1.42.6 (core 1.131.9)
were processed in the wrong order:
INBOX UIDs on the second message are resent messages not moved due to #5155, this bug was fixed in core 1.134.0.
The problem is that message 1 with UID 193 was processed later than message 2 with UID 194. Second message got messageId 224 (this is a database row ID) and first message got messageId 234.
This reordering is unlikely to be due to partial message download because it is disabled by default and because
rcvd_timestampof messages is too close.Delta Chat sorts messages by INTERNALDATE:
deltachat-core-rust/src/imap/session.rs
Line 157 in 76bbd5f
testrun.org runs Mailcow (Dovecot), so it looks like INTERNALDATE gets changed when a batch of messages are moved to DeltaChat folder.
The text was updated successfully, but these errors were encountered: