Silence XML plaintext backup conversion? #121

JoeGretch · 2023-07-19T02:30:34Z

Thank you for creating such a useful app.

Would you be willing to write a script that will convert the Silence XML plaintext backup file into a JSON file to be imported?

Here is a sample from the Silence backup:

<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<!-- File Created By Silence -->
<smses count="12972">
 <sms protocol="0" address="+15557639101" date="1606237355859" type="1" subject="null" body="I can&apos;t fri but I can Thur I don&apos;t need assistance picking it up and can come by tonight and pay if that works" toa="null" sc_toa="null" service_center="null" read="1" status="-1" locked="0" />
 <sms protocol="0" address="+15557639101" date="1606237237747" type="2" subject="null" body="Hi Tracy. The boiler is all disconnected and ready to be lifted from it&apos;s pad. Can you come in Friday some time to load it up? I have to leave on Saturday and will be away for a week, and will also be tied up all day on Thanksgiving.&#10;&#10;Please let me know if that will work out for you.&#10;&#10;Thanks!" toa="null" sc_toa="null" service_center="null" read="1" status="-1" locked="0" />
 <sms protocol="0" address="+15557639101" date="1606077873112" type="1" subject="null" body="Yes all good thx I think I can hook every thing up now that I&apos;ve looked at it hope all is well. With y&apos;all also" toa="null" sc_toa="null" service_center="null" read="1" status="-1" locked="0" />
 
                                     . . . . . . .
 
  <sms protocol="0" address="5556637843" date="1689601387150" type="1" subject="null" body="We are on our way to get gas and head out" toa="null" sc_toa="null" service_center="null" read="1" status="-1" locked="0" />
</smses>

Thank you again.

The text was updated successfully, but these errors were encountered:

tmo1 · 2023-07-20T19:37:32Z

I'm not promising to do it, but I'll consider it. The format doesn't look too complicated. Are there MMS messages, or only SMS?

JoeGretch · 2023-07-20T20:27:56Z

I do not think Silence exports MMS successfully. There certainly are no image or other files associated with these messages.

Thank you. I appreciate your just looking at it.

Addresses: #121

tmo1 · 2023-07-26T03:01:33Z

A conversion script is now available here (documented here). It works on the sample XML you provided, but please test and report back here.

JoeGretch · 2023-07-26T13:26:46Z

Thank you!

Unfortunately, emojis, seem to break the Element Tree parser. There weren't any emojis in the sample I sent you; I apologize. This is the result I got:

xml.etree.ElementTree.ParseError: reference to invalid character number: line 12, column 93

and this is line 12 from my file:

<sms protocol="0" address="+15557639101" date="1601816103259" type="2" subject="null" body="&#55357;&#56397;" toa="null" sc_toa="null" service_center="null" read="1" status="-1" locked="0" />

I might just manually strip out the lines with emojis and see if I can make the file work that way, but I wanted to let you know how the initial test went.

Thank you again.

tmo1 · 2023-07-26T14:05:14Z

I looked into this, and it turns out that the problem is that Silence is emitting invalid XML - this was first reported seven years ago! You can try the entityfixer code posted by Calvin-L in that thread.

DavidHenryThoreau · 2023-11-09T15:12:16Z

@tmo1 Can you implement entityfixer in the https://github.com/tmo1/sms-ie/blob/master/tools/silence-convert.py ?

DavidHenryThoreau · 2023-11-09T16:28:26Z

./silence-convert.py SilencePlaintextBackup.xml
Traceback (most recent call last):
  File "/tmp/l/./silence-convert.py", line 46, in <module>
    tree = ET.parse(input_file)
           ^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/xml/etree/ElementTree.py", line 1218, in parse
    tree.parse(source, parser)
  File "/usr/lib/python3.11/xml/etree/ElementTree.py", line 580, in parse
    self._root = parser._parse_whole(source)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
xml.etree.ElementTree.ParseError: reference to invalid character number: line 75, column 108

Silence produces invalid XML when encoding certain characters (such as emojis). The Silence XML fixer tool, based on Calvin-L's entityfixer() function and the comments to his Gist, fixes the XML: https://git.silence.dev/Silence/Silence-Android/-/issues/317 https://gist.github.com/Calvin-L/5232f876b8acf48a216941b8904632bb Addresses: #121

DavidHenryThoreau · 2023-11-10T15:07:20Z

Nice feature, SMS are imported with emoji 👍

Do you think MMS can be save and restore ?
SMS groups (2 or 3 people are not restored) -> they're not in SilencePlaintextBackup.xml

tmo1 · 2023-11-10T16:39:22Z

Nice feature, SMS are imported with emoji 👍

I'm glad it's working!

Do you think MMS can be save and restore ?
SMS groups (2 or 3 people are not restored) -> they're not in SilencePlaintextBackup.xml

We can obviously only restore data that's present in the XML file. If there's anything there that isn't being converted / restored, please post some examples and I'll take a look.

DavidHenryThoreau · 2023-11-11T18:00:34Z

We can obviously only restore data that's present in the XML file. If there's anything there that isn't being converted / restored, please post some examples and I'll take a look.

MMS are stored in /data/data/org.smssecure.smssecure/app_parts/part1001758286944762215.mms

These files are encrypted :

file part1001235047216594581.mms
part1001235047216594581.mms: data

tmo1 · 2023-11-11T23:38:56Z

Where do those files come from? What's the encryption scheme, and where are the keys? If you provide sample files and keys, I can take a look (if you're willing expose potentially private data and metadata), but there's no way I can do anything without that.

DavidHenryThoreau · 2023-11-12T08:42:17Z

Where do those files come from? What's the encryption scheme, and where are the keys? If you provide sample files and keys, I can take a look (if you're willing expose potentially private data and metadata), but there's no way I can do anything without that.

I'd like to decrypt a file first , in order to see what file it's and then post it.

Here's the MMS database structure :
https://git.silence.dev/Silence/Silence-Android/-/blob/master/src/org/smssecure/smssecure/database/MmsDatabase.java

The keys to decrpyt are located in :
org.smssecure.smssecure/shared_prefs/SecureSMS-Preferences.xml

tmo1 · 2023-11-26T00:15:32Z

Thanks. I'm not planning to comb through the code to figure out the database structure and encryption scheme, but if you make progress on it, please post your results here.

dngray · 2023-12-06T06:43:10Z

I created a tool to do this in python to txt format https://github.com/dngray/silence-backup-parse/blob/main/silence_backup_parse.py there were some specifics regarding emojis.

as for MMS those are not exported in the XML file.

DavidHenryThoreau · 2023-12-06T07:13:18Z

@dngray Nice, emoji and SMS are well exported by using @tmo1 source code
What is missing :

SMS with more than 2 people (which is not present in the XML)
MMS see my previous messages

dngray · 2023-12-06T09:30:22Z

SMS with more than 2 people (which is not present in the XML)

That's probably because those are actually MMS messages and not SMS.

DavidHenryThoreau · 2023-12-06T09:51:05Z

Yes, I don't have found what kind of algo is used to encrypt MMS files :
/data/data/org.smssecure.smssecure/app_parts/part1001758286944762215.mms
Do you have any ideas about these files ? How to decrypt them ?
Keys are locate in :
org.smssecure.smssecure/shared_prefs/SecureSMS-Preferences.xml

tmo1 · 2023-12-06T14:20:23Z

That's probably because those are actually MMS messages and not SMS.

Correct. SMS messages cannot have more than one recipient - messages sent to more than one recipient are automatically sent as MMS messages, even when they're only simple text messages with no "media" involved.

Pommede · 2023-12-07T10:03:55Z

Hi
sure i'm a stupid
but i tried
python silence-convert.py SilencePlaintextBackup.xml
and i've got:
Traceback (most recent call last):
File "/home/user/silence-convert.py", line 46, in
tree = ET.parse(input_file)
^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/xml/etree/ElementTree.py", line 1218, in parse
tree.parse(source, parser)
File "/usr/lib/python3.11/xml/etree/ElementTree.py", line 580, in parse
self._root = parser.parse_whole(source)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
xml.etree.ElementTree.ParseError: reference to invalid character number: line 4, column 116

then i did like recommended:

python silence-xml-fixer.py SilencePlaintextBackup.xml SilencePlaintextBackupfixed.xml
and i've got nothing happened..after 15min...i quit..

and
python3 silence_backup_parse.py SilencePlaintextBackup.xml

and i've got a SilencePlaintextBackup1.zip (as i wrote in the python file) but empty file....

somebody can help me ??? :-(
thks !

DavidHenryThoreau · 2023-12-07T10:07:19Z

@Pommede https://github.com/tmo1/sms-ie/blob/master/tools/Tools.md#silence-convertpy

Pommede · 2023-12-07T10:10:49Z

that's what i did, (first command) or something wrong on my command?

DavidHenryThoreau · 2023-12-07T10:12:58Z

xml.etree.ElementTree.ParseError: reference to invalid character number: line nnn, column mmm

If this error is encountered, first use the XML fixer tool to produce valid XML:

silence-xml-fixer.py < silence-xxx.xml > silence-xxx-fixed.xml

then run the converter on the fixed XML:

silence-convert.py <silence-xxx-fixed.xml>

Pommede · 2023-12-07T10:14:46Z

thank for your response but i did all that (it's my text.. :-()

DavidHenryThoreau · 2023-12-07T10:17:24Z

You've missed < and >

Pommede · 2023-12-07T10:24:05Z

yeah the first command worked
python silence-xml-fixer.py <SilencePlaintextBackup.xml> SilencePlaintextBackupfixed.xml
but the second
no <> if i put those i've got
fish: Expected a string, but found end of the input

but python silence-convert.py SilencePlaintextBackupfixed.xml
i've got a zip with something....i'll try to another sms app
thks

Pommede · 2023-12-07T10:32:49Z

it's workin !! :-)
many thks.
no mms but it's a silence issue ?

DavidHenryThoreau · 2023-12-07T10:34:05Z

MMS are not located in the XML still working about it

Pommede · 2023-12-07T10:38:13Z

oki, very good job, i was afraid to be stuck with silence rrrr

DavidHenryThoreau · 2023-12-14T08:02:28Z

I'm looking at org.smssecure.smssecure/shared_prefs/Secure/SMS-Preferences.xml it mentions curve25519

Rapid search can lead to :
https://github.com/topics/curve25519
The rage project may can decrypt file ??

-d, --decrypt Decrypt the input.
https://github.com/str4d/rage#passphrase-protected-identity-files
https://github.com/str4d/rage

tmo1 · 2023-12-14T13:09:03Z

Silence / SMSSecure uses the Signal encryption protocol, which uses / can use curve25519 in its operation. I think that this may be only applicable to the encryption used for messages in transit, though, and not to the encryption used for the on-disk message databases.

There are various tools for decoding the encypted backups produced by Signal, including signalbackup-tools and signal_for_android_decryption, but they are not designed to be used with the raw database files retrieved from the Android filesystem.

DavidHenryThoreau · 2024-01-09T10:26:17Z

There's also ; that may be needs to be adapted to Silence
https://github.com/alexlance/signal-sms-mms-importer

tmo1 added the enhancement New feature or request label Jul 20, 2023

tmo1 added a commit that referenced this issue Jul 21, 2023

[messages] Add Silence XML converter

7600aac

Addresses: #121

fantasymeal mentioned this issue Aug 26, 2023

Trouble importing messages #123

Closed

DavidHenryThoreau mentioned this issue Nov 9, 2023

How to import JSON hydrargyrum/breakthesilence#1

Closed

tmo1 self-assigned this Nov 10, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Silence XML plaintext backup conversion? #121

Silence XML plaintext backup conversion? #121

JoeGretch commented Jul 19, 2023 •

edited

tmo1 commented Jul 20, 2023

JoeGretch commented Jul 20, 2023

tmo1 commented Jul 26, 2023

JoeGretch commented Jul 26, 2023

tmo1 commented Jul 26, 2023

DavidHenryThoreau commented Nov 9, 2023

DavidHenryThoreau commented Nov 9, 2023

DavidHenryThoreau commented Nov 10, 2023 •

edited

tmo1 commented Nov 10, 2023

DavidHenryThoreau commented Nov 11, 2023

tmo1 commented Nov 11, 2023

DavidHenryThoreau commented Nov 12, 2023

tmo1 commented Nov 26, 2023

dngray commented Dec 6, 2023

DavidHenryThoreau commented Dec 6, 2023

dngray commented Dec 6, 2023

DavidHenryThoreau commented Dec 6, 2023

tmo1 commented Dec 6, 2023

Pommede commented Dec 7, 2023 •

edited

DavidHenryThoreau commented Dec 7, 2023

Pommede commented Dec 7, 2023 •

edited

DavidHenryThoreau commented Dec 7, 2023

Pommede commented Dec 7, 2023

DavidHenryThoreau commented Dec 7, 2023

Pommede commented Dec 7, 2023

Pommede commented Dec 7, 2023

DavidHenryThoreau commented Dec 7, 2023

Pommede commented Dec 7, 2023

DavidHenryThoreau commented Dec 14, 2023 •

edited

tmo1 commented Dec 14, 2023

DavidHenryThoreau commented Jan 9, 2024

Silence XML plaintext backup conversion? #121

Silence XML plaintext backup conversion? #121

Comments

JoeGretch commented Jul 19, 2023 • edited

tmo1 commented Jul 20, 2023

JoeGretch commented Jul 20, 2023

tmo1 commented Jul 26, 2023

JoeGretch commented Jul 26, 2023

tmo1 commented Jul 26, 2023

DavidHenryThoreau commented Nov 9, 2023

DavidHenryThoreau commented Nov 9, 2023

DavidHenryThoreau commented Nov 10, 2023 • edited

tmo1 commented Nov 10, 2023

DavidHenryThoreau commented Nov 11, 2023

tmo1 commented Nov 11, 2023

DavidHenryThoreau commented Nov 12, 2023

tmo1 commented Nov 26, 2023

dngray commented Dec 6, 2023

DavidHenryThoreau commented Dec 6, 2023

dngray commented Dec 6, 2023

DavidHenryThoreau commented Dec 6, 2023

tmo1 commented Dec 6, 2023

Pommede commented Dec 7, 2023 • edited

DavidHenryThoreau commented Dec 7, 2023

Pommede commented Dec 7, 2023 • edited

DavidHenryThoreau commented Dec 7, 2023

Pommede commented Dec 7, 2023

DavidHenryThoreau commented Dec 7, 2023

Pommede commented Dec 7, 2023

Pommede commented Dec 7, 2023

DavidHenryThoreau commented Dec 7, 2023

Pommede commented Dec 7, 2023

DavidHenryThoreau commented Dec 14, 2023 • edited

tmo1 commented Dec 14, 2023

DavidHenryThoreau commented Jan 9, 2024

JoeGretch commented Jul 19, 2023 •

edited

DavidHenryThoreau commented Nov 10, 2023 •

edited

Pommede commented Dec 7, 2023 •

edited

Pommede commented Dec 7, 2023 •

edited

DavidHenryThoreau commented Dec 14, 2023 •

edited