Handle RCS messages properly, and improve group MMS #1097

locofocos · 2023-04-23T09:55:16Z

This fixes RCS importing 🎉

It also improves how group MMS threads are imported. Basically it includes all the recipients in the "to" field, and puts everyone's name in the subject, so that gmail does a much better job mimicking the group threads on your phone.

I've been toying around with it on my Pixel 3a running Android 10. I've tested with a mix of SMS, group MMS chats, and RCS chats.

I'm completely open to the idea that I was a little heavy-handed with my refactor. I ran with what made sense. I'm fine if someone else wants to take this as as proof-of-concept and run in a slightly different direction to make this mergeable.

TODO

See if there are any unit tests that have been broken
Probably some new unit tests (to be honest..... I really don't want to build these out).
Give this another look over for general code quality. The old version of Java (and me being a bit rusty with Java) made some things like sets/arrays/uniq/etc a bit clunky.

Testing

Tips for testing:

If you want to try running this multiple times and tweaking how we generate ids/subjects/etc, make sure to delete your emails generated from testing AND empty gmail's trash. Otherwise gmail will cache things. I saw it caching the REFERENCES field when I re-used the same MESSAGE_ID.
Once you've identified some messages on your phone for a certain test case, it can be helpful to just process those messages. You can hardcode timestamps in the queries in app/src/main/java/com/zegoggles/smssync/service/BackupQueryBuilder.java. https://www.epochconverter.com/ is your friend here.

Some test cases that I've gone through on my phone, and verified that the backup looks correct in Gmail:

RCS message with 1 person. working as of bec0a00
RCS message with multiple people. Working as of bec0a00
MMS thread with lots of people going back and forth. working as of bec0a00
Message with 1 person, both SMS and MMS (texting a non-RCS user, but sometimes sending images).
- Less than 100 messages. working as of bec0a00
- At least 100 sms messages before the MMS got backed up. Not working
  - Unfortunately we'll probably have to live with this limitation. In theory it could be fixed, but it would require processing SMS and MMS messages interleaved, from oldest to newest. Those are pulled from two separate data sources and two queries, see BackupItemsFetcher.getItemsForDataType and Consts.MMS_PROVIDER / SMS_PROVIDER. So we would have to build some intermediate caching mechanism that gathers all the SMSs, gathers all the MMSs, interleaves them, then feeds them into the existing logic.
  - I tried using the "only backup 100 at a time" setting. Unfortunately it looks like this processes batches of SMSs first, then starts processing MMSs once the SMSs are exhausted. So it doesn't help.

Adopting this new version

Here is my 2 cents for how affected users could get their backups into a clean-ish state:

The "I don't care" way: just start using the new version, and everything going forwards will be fixed.
The lazy way: just reset the backup state, back up everything again, and deal with duplicates in gmail.
The harder way: Go into Gmail and update all existing emails with label old-backups. Download the new app version. Reset the backup state and backup everything again (to your favorite label, like SMS). For whatever time period you still have on your phone, delete emails from that time period with label old-backups. Verify that you have a nice continuity when looking at both old-backups and SMS. Then change all the old-backups emails over to the SMS label.

…droid Studio (everything else)

…, plus the subject.

Sometimes you would see both of these subjects in gmail: 123-123-1234/Dad 123-123-1234/Dad/Dad

…reads, due to grabbing the contact ID from a random person in each thread

kurahaupo · 2023-04-23T10:32:32Z

@locofocos Thank you so much for doing this; it will close a large number of outstanding tickets.

I will review in detail shortly.

(It will still be up to @jberkel to publish to Google Play.)

kurahaupo · 2023-04-23T10:41:50Z

I'm really happy with the separation of records into sender and recipients.
This will fix not just RCS, but almost every issue in the "half missing" category.

Again, many thanks.

locofocos · 2023-04-23T16:55:54Z

app/src/main/java/com/zegoggles/smssync/mail/MessageGenerator.java

+        // Tip - if you want to try different strategies for computing mmsThreadId against the same messages,
+        // make sure to generate a new MESSAGE_ID as well. It seems like gmail caches REFERENCES if you reuse the same MESSAGE_ID.


Some context, from what I learned doing this: MESSAGE_ID is the unique message on gmail's end. REFERENCES is the field (on gmail's end) that determines which email an email is replying to.

For the last message grouping fix I built, I wasn't seeing it take effect, even after I deleted all my sms backups from gmail and ran the back again. I was able to determine that gmail was caching the REFERENCES field by clicking "show original" in gmail and looking at the message header. I had to force the app to generate new MESSAGE_IDs. Basically there's some string we pass into the hash function that generates MESSAGE_ID. You can just concatenate "v2", "v3", etc. to the hash input. It's almost like bumping a version number. Once I generated a new MESSAGE_ID for those messages I had already backed up before, then my last fix started working.

However, this has repercussions for releasing this fix. I tried backing up all the messages on my phone with a clean build of this current PR. I'm using a different gmail label (SMS-testing instead of the usual SMS) to keep things clean, but the app was still generating the same MESSAGE_ID as before. So when I checked on some older (already backed up) messages, they weren't grouped correctly 😢

I can vet this out, but I believe if we version/change the hash input (like we start concatenating "v2"), that will force gmail to recognize these as new messages. So then users will be able to reset the app state and re-backup their existing messages, and then older threads should get fixed 🎉 I expect the old messages would linger around in gmail as well, so there may some duplicates... but users will only get this if they choose to re-backup their existing messages.

This will also be a useful opportunity for me to check against more test messages. I'll check this out some day soon!

Good news! Versioning the hash input ab80cc0 fixes this problem nicely. This will be helpful to force gmail to recognize these messages as new threads and group them correctly.

locofocos · 2023-04-23T16:57:00Z

@kurahaupo you're welcome! Thanks for the initial review. I'll get this cleaned up and tested further some time soon, I hope.

…MS (some messages have images)

…s not broken into 2 threads

…is worked before, but now avoids a warning log

locofocos · 2023-04-26T05:50:13Z

@kurahaupo I've given this some solid testing on my phone, and I'm happy with the result.

Is the travis CI build running against this PR? I was looking for the build output so I could at least see which tests I need to update.

edit - I might need some some more time to look over the results and debug more. All of those test cases look good. But I did a another huge backup last night, and I'm seeing some ungrouped messages.

kurahaupo · 2023-04-26T22:27:10Z

I'll check on the CI when I get home to my PC

hubono · 2023-09-02T20:09:52Z

Hi
I tried to compile from source but it keeps failing. Can someone post an APK please

locofocos · 2023-09-02T23:35:38Z

@hubono I debated whether I should post a copy of my latest APK build, just in case life happens and we don't get this PR merged.

Caveats, if this wasn't obvious:

This code hasn't been well reviewed
It's code that some stranger on the internet wrote
It's built in debug mode, with a self-signed certificate (something like that). So android won't allow you to upgrade your existing app with this. Android is rightly protecting you from installing an APK built by someone other than the author. So you'll have to uninstall your existing app first.

However, I've been using it for months. No promises, but I'm happy with the results. So here is the APK I built locally from ac71d55

SMS backup plus with RCS fix v2.apk.zip

locofocos added 5 commits April 22, 2023 23:46

Get it compiling both via CLI (JavaVersion.VERSION_1_8 change) and An…

1bfc09d

…droid Studio (everything else)

Get RCS working

1643bdf

Fix buggy behavior with group texts. Include everyone in to/from list…

e512041

…, plus the subject.

Mostly fix bug, where a single thread would get multiple gmail threads

35dd9ec

Sometimes you would see both of these subjects in gmail: 123-123-1234/Dad 123-123-1234/Dad/Dad

Fix bug more fully, where a single thread would get multiple gmail th…

d9ef19f

…reads, due to grabbing the contact ID from a random person in each thread

locofocos mentioned this pull request Apr 23, 2023

Backup to Gmail only backing up one half of conversation #1055

Open

locofocos commented Apr 23, 2023

View reviewed changes

locofocos added 4 commits April 24, 2023 23:03

Version the MESSAGE_ID hash generation so users can fix old messages

ab80cc0

Fix grouping for case of message thread with 1 person, both SMS and M…

26fc10f

…MS (some messages have images)

Fix bug introduced by last commit. Ensure RCS message with 1 person i…

bec0a00

…s not broken into 2 threads

Cleanup. Parse addresses listed as "CC" for incoming RCS messages. Th…

ac71d55

…is worked before, but now avoids a warning log

locofocos marked this pull request as draft April 26, 2023 14:15

kurahaupo marked this pull request as ready for review April 26, 2023 22:26

opello mentioned this pull request Apr 8, 2024

SMS Backup+ removed from Google Play Store #1104

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Handle RCS messages properly, and improve group MMS #1097

Handle RCS messages properly, and improve group MMS #1097

locofocos commented Apr 23, 2023 •

edited

kurahaupo commented Apr 23, 2023

kurahaupo commented Apr 23, 2023

locofocos Apr 23, 2023

locofocos Apr 26, 2023

locofocos commented Apr 23, 2023

locofocos commented Apr 26, 2023 •

edited

kurahaupo commented Apr 26, 2023

hubono commented Sep 2, 2023

locofocos commented Sep 2, 2023

		// Tip - if you want to try different strategies for computing mmsThreadId against the same messages,
		// make sure to generate a new MESSAGE_ID as well. It seems like gmail caches REFERENCES if you reuse the same MESSAGE_ID.

Handle RCS messages properly, and improve group MMS #1097

Are you sure you want to change the base?

Handle RCS messages properly, and improve group MMS #1097

Conversation

locofocos commented Apr 23, 2023 • edited

Testing

Adopting this new version

kurahaupo commented Apr 23, 2023

kurahaupo commented Apr 23, 2023

locofocos Apr 23, 2023

Choose a reason for hiding this comment

locofocos Apr 26, 2023

Choose a reason for hiding this comment

locofocos commented Apr 23, 2023

locofocos commented Apr 26, 2023 • edited

kurahaupo commented Apr 26, 2023

hubono commented Sep 2, 2023

locofocos commented Sep 2, 2023

locofocos commented Apr 23, 2023 •

edited

locofocos commented Apr 26, 2023 •

edited