How to use Apple export script to import to Thunderbird

[johnstonesnow]

I have a load of local folders in Apple Mail (NOTE: It's running on OSX 10.14 Mojave, old style)i. exported these from Mac Mail which took a few hours at least.

I have followed instructions to import into TB via this plugin. Tried every option possible but fails every time with message: "Import Done 0/0"

Any ideas appreciated.

[cleidigh]

@johnstonesnow
What explicitly are you importing, a single mbox or a structure produced from the script we discussed?
@cleidigh

[johnstonesnow]

I exported all my local mail using Mac Mail's export mailboxes function. So I have a nested tree structure with folders and mboxes.
I haven't (couldn't!) use any 'scripts'.
I read somewhere that because I am using a very old OS (Mojave), the Mail export should work with this plugin.

[cleidigh]

@johnstonesnow
As far as I have seen, nothing is different with Mojave.
For apple structured exports you have to transform the structure to Thunderbird structure with the script. IETNG can then import.
@cleidigh

[johnstonesnow]

Ok, that would explain it. But I have no idea what script, I know the thread you're referring to but I can't follow it now it's gone into technical stuff. I am not a dev, but even if I was, I thought the last I read was that the script is broken and nobody has time to fix it, is that right or is there something working that you think it might be worth trying to learn how to run?
Thanks

[cleidigh]

@johnstonesnow
No problem, I will back up.
Here is the script and instructions
https://github.com/thunderbird/import-export-tools-ng/wiki

The most important thing is to make a backup of your export because the script modifies the tree in place. Since it might take a couple of tries you want the backup to start over.
We thought the script might have a problem, but that was a user issue. I did a small transform on Windows.
@cleidigh

[johnstonesnow]

Thanks. Yes I took a zip backup to burn and put in safe for keeps. Let's see if I can work out how to follow these instructions!

[cleidigh]

@johnstonesnow
You will want your backup on your system ready to recopy.
First thing you want to do is run the script with no arguments just to see python and the script are good.. Next try using the path to the top of the export. That directory should have a subdirectory with a .mbox suffix.
Report the resulting structure.
Christopher

[johnstonesnow]

Hmm, as suspected, it's not for average folks, dev-types only! but I am trying.
I checked with "python" command in terminal and i have python 2 running, so I am downloading a Python3 installer.

But I will be stuck when i get that installed.

"An experimental script to restructure an Apple Mail export, applembox2sbd.py, is available in the utility-scripts folder of this repository."

I can see a page of code at the link, but what do I do with it?

Thanks

[cleidigh]

download the script with the raw link.
Once you have python 3, you should be able to run it.
Christopher

[johnstonesnow]

@johnstonesnow You will want your backup on your system ready to recopy. First thing you want to do is run the script with no arguments just to see python and the script are good.. Next try using the path to the top of the export. That directory should have a subdirectory with a .mbox suffix. Report the resulting structure. Christopher

Thanks, but I hardly understood a word. I have backup (duplicate folder) of my exported tree.
No idea how to run a script, don't know what arguments are, sorry!

[johnstonesnow]

I clicked the "raw" button and got a full page of the code/script

[johnstonesnow]

do I copy that and paste into terminal or something?

[johnstonesnow]

Hmm. I installed the python3 installer package. it completed successfully. To confirm I went to terminal and typed "python" again, hoping to see V3 installed but it still says V2:

"Python 2.7.10 (default, Feb 22 2019, 21:55:15)
[GCC 4.2.1 Compatible Apple LLVM 10.0.1 (clang-1001.0.37.14)] on darwin
Type "help", "copyright", "credits" or "license" for more information."

Do I need to reboot for it to take effect?

[johnstonesnow]

I have to run out but will be back in an hour or two to make some attempt at running this script. Just bear in mind I have no clue about making directories in terminal, or even finding a directory. Only way I have ever got a file location into terminal is by dragging and dropping a finder folder at the end of a command.
I will have a go later on but any dumba55-friendly tips would be appreciated :)

[cleidigh]

yes
copy and save as the script name

hmm I don't know how to make python 3 the default.
Did the installer mention anything on this?
I don't want to mess up your current system especially since you are not a developer.
The other possible approach is to use your Linux target which probably has python 3
Christopher

[cleidigh]

just looked.
you should try python3
that should invoke version 3

[cleidigh]

@johnstonesnow
Ok, we will do our best.
I wish I had a solution that was simpler, but it's the nature of a lot migration scenarios.
I think we can do it, I will do my best. Once you try the python3 command we will go from there.
Not everyone is a developer, no dumba55 required %-)
Christopher

[johnstonesnow]

hehe, thanks.
Re messing up this machine, no worries there. It's my last iMac (a very old one that's being used for zeroing my rifle once this job is done, and no I am not joking).

python3 - that did it :) ...

"Python 3.12.2 (v3.12.2:6abddd9f6a, Feb 6 2024, 17:02:06) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information."

So... "copy and save as the script name".
By that do you mean use the browser menu to "save as html" to save to my computer then rename the file "apple2mboxsbd.py" ?

[cleidigh]

@johnstonesnow
I'm on East coast time, but usually online by 12:00. I will be online until around 6:30. Let me know when you want to "live walk through"
In the meantime I will post some steps and notes. I will also fire up my MacAir which I think has Mojave also.
Christopher

[johnstonesnow]

Hi, that's extremely kind of you. I am sitting down for an hour (minimum) now.
I am hopeful it won't take too much time, just need to learn how to run the script. I do have Python3 installed now, so that's a step in the right direction. I have the Mac Mail export (with duplicates) ready to try a conversion.
thanks very much

[cleidigh]

@johnstonesnow
Sorry, Just got moved to bed, I'm disabled with ALS so do programming by eyes and don't have a lot of time now. If you are still on we can do a bit. I would like you to create a new directory right under your home directory for ease. Name it mboxcvt. We will download the script there next. We can do these first few steps with Finder then we will have to use the terminal.
C

[johnstonesnow]

Hi Christopher, sorry to hear that. I am in EU so it's the small hours now and usually that'd be fine but I have to be up early tomorrow and can't adjust it. Can I make a suggestion, especially given you dont have a lot of time...

Back and forth one command at a time may be ideal to avoid mistakes but I suspect it's unnecessary. IF (and only if) you have the time, maybe you could just give me some bullet point instructions and I think I will be able to follow those when I get back tomorrow lunch time.

Whilst I am no developer, I do have a bit of experience with Mac (terminal and finder stuff in Library etc), I certainly know more than the average Mac user.

I can create a directory under home, I have created one called mboxcvt alongside movies, pictures etc, in user home folder which i assume is what you mean.

Hopefully you can drop some steps for me to follow beyond that and I will give it a whirl tomorrow. Dont worry about mistakes my end, worst case I have a TimeMachine drive backing up every hour so I can always roll my system back but I am sure that won't be necessary. i have run a script or two before over the years, as well as adjusting one or two files in Terminal (such as to stop the damn update notifications!)

I have to run now but really grateful for your kindness helping me here. Drop me some tasks and let's see if I can impress myself (if not anybody else :D)

Thank you
Mark

[cleidigh]

Ha, you are supposed to be John (Snow of Game of Thrones?) %-)
I will give some more steps, but will need some interaction after. I also have to be brif as typing by eyes is onerous.

Next step, download script:

• Click script link and look for download button on top right,, box with down arrow in it
• Should download to your download dir, then move to our mboxcvt dir
• Now open a terminal
• Change to the home dir with
  cd ~
• change to mboxcvt
  cd mboxcvt
• See if the script runs
  python3 applembox2sbd.py
• you should see it gripe about not having a IN argument (parameter to cmds)

Setting up other directories, you can do regardless of above working

• Under mboxcvt make two directories exptree and exporig
• copy your export tree into both directories
• Explain the the top levels of directories in exptree

We will have to interact tomorrow any time after 12 Eastern time.
BTW I always love to know where my users are if you don't mind saying...
Christopher

[johnstonesnow]

Yes, well spotted, but you're forgetting I have to do sentry duty to watch out for white walkers :-D
Unfortunately for me I'm in the UK. Typing by eyes, that must be quite tiring. Will do my best to keep this as short (your end) as possible.

OK, step by step....

- Click script link and look for download button on top right,, box with down arrow in it

Ok, by "script link" I assume you mean the one that goes to this page - https://github.com/thunderbird/import-export-tools-ng/blob/master/utility-scripts/apple2mboxsbd.py
Assuming so, I have clicked the download button (one with tooltip: "Download Raw File").

- Should download to your download dir, then move to our mboxcvt dir

I downloaded it straight into mboxcvt dir

• **Now open a terminal
• Change to the home dir with
• cd ~
• change to mboxcvt
• cd mboxcvt
• See if the script runs
• python3 applembox2sbd.py**

ha, no such file or directory. Get this, I remembered one of about 3 commands I know (maybe I was lying when I said I wasn't a dev :) !) and did a "ls" to list contents. Then realised you put the 2 in the wrong place :D - So I ran:

python3 apple2mboxsbd.py

• you should see it gripe about not having a IN argument (parameter to cmds)

Yep! Output:

"usage: apple2mboxsbd.py [-h] [--verbose] [--dry-run] IN
apple2mboxsbd.py: error: the following arguments are required: IN"

• **Setting up other directories, you can do regardless of above working
• Under mboxcvt make two directories exptree and exporig**

I am assuming I can do this in Finder so will do it that way. I think I remember mkdir being the command but lack of confidence leads me to Finder wherever possible.

Ok, now we're cooking on gas :)
Here's Finder now:

Ok, these last two bullets I got a bit stuck with.

- copy your export tree into both directories

Assuming I know what you mean but want to be 100% sure before I do anything further. by "export tree", am i right in thinking you mean the entire folder full of exported folders from Mac Mail? If so, you mean my folder called "mailboxes export" which is on my desktop, looks like this:

So should I select all contents OF 'mailboxes export', and copy those into the two folders we created?
I guess there's no harm in going ahead and doing that anyway, can always delete.
This is done, now both of those folders look like this:

- Explain the top levels of directories in exptree

Wasn't sure but maybe you just mean for me to show what they look like, hopefully done that above.

If my assumptions are correct, I think we are now at the point you hoped I'd be. :)

Thank you again, sorry for all the typing but seems easier for you to read (every detail of my actions) rather than me asking and waiting for you to answer, and have to type more needlessly.

[cleidigh]

@johnstonesnow
Excellent, you were under selling yourself!
So now we have our setup ready for converting.
from terminal, get to the mboxcvt dir
Then:
python3 apple2mboxsbd.py ./extptree

Then screenshot the exptree directory, with luck we will have our transform.
Christopher

[johnstonesnow]

Finished!
(I am sure this doesn't need saying but just to be completely thorough, i removed the extra "t" from the command above "./extptree"
It ran and "finished!"

I watched Finder live as it happened, like magic. :)

Old on left, New on right

Looks like it worked to my untrained eye.

25 items in each folder
The old folders are still folders but with .sbd appended.
The old .mbox files have the same name but the .mbox has been removed.

Are we looking good?

thanks so much for this Christopher

[cleidigh]

@johnstonesnow
Fantastic, no problem!
Now the tree can be imported. Do you want to do that on Linux?
C

[johnstonesnow]

Great!
That was going to be my next question.
Since this is all about moving to Fedora, would it be best to run Thunderbird on the Fedora machine (much more powerful machine), move these files over with USB stick and run the import there? I can run on Mac, whichever you think is best, the only consideration I know of is the performance difference. This machine is an old dog, the Fedora one is a new top spec AMD thinkpad.
What say you Mr C? :)

[johnstonesnow]

Yes it wouldn't surprise me to find the error was on the mac side

bash: out.txt: command not found...

[johnstonesnow]

Please don't overdo it on this, no pressure as i have these emails backed up and accessible on the imac for the foreseeable future

[johnstonesnow]

Well the re-index of Mac Mail failed. Gonna have to work on clearing some room, this SSD is so full. Will work on that.
Interestingly, when I checked trash (first place i check to clear space on SSD) I see mbox files in there, and some funny names/appendices to folders. Not honestly sure why there is any of these in trash at all, I don't remember deleting any. Wondered if it's part of the conversion script's process? Just curious.

Here's a screenshot of a few of them in trash, no idea where those numbers on folder names came from!

[cleidigh]

Afternoon...
Nope ietng wont create anything.
The From count from grep is clear, the Mac mail counts are not matching what the export is. I would be interested in the grep on your senf mbox.
Anyway, what would you like to do further?

[johnstonesnow]

Hi Christopher.
well I'd like to get my emails from Mac to Thunderbird on Linux :)
From what you said there, am I right in thinking you believe the export/conversion/import happened properly, it's just the Mac Mail reporting incorrect numbers of msgs in folders? If so, I guess I can confirm that by manually counting them. If that's the case I guess I'm good to go, if I can get some confidence in that it would be good!
Did you say there's a new version of the script? I am wondering about doing this, to try to gain confidence that I have all the emails:

1. Re-index Mac Mail - if folder counts are off, that is the fix so should solve it.
2. Reconvert it with your delightful script :)
3. Clear down TB and reimport from new conversion

What are your thoughts on this please?

[cleidigh]

I would definitely do the rebuild and reindex. Then do the import again and we can see if there are any problems after that.
Christopher

[johnstonesnow]

Roger that, thanks

[johnstonesnow]

Just a thought... Is there any chance that grep count command would work on Mac? If so, I could try counting the Mac Mail exported folders just to see what numbers come back in terminal

[johnstonesnow]

I missed your comment about being interested in grep count for Sent folder. Here's what I reported earlier for Mac v TB:

Mac - 37,374 msgs
TB - 5116 msgs

Here's what that grep count command returned for Sent folder: 5176

Looking like you're right about wonky Mac counts!

[johnstonesnow]

I just looked at folder properties on two folders:

1. The folder on Mac where i put the entire Mac Mail export, before conversion.
2. The exptree folder on Linux which I brought over after conversion

9.81GB (Mac) and 9.8GB (Fedora) - suggests all data came over, so any failure would have to be Mac Mail's fault i think.

Idea - I do have another more powerful iMac (used for business stuff). Just wondering whether I could take the Mac Mail export I have, via USB stick, to the more powerful iMac, and create a temp new user account to try importing it into Mac Mail there. Wouild be interesting to see if folder counts are different

[johnstonesnow]

WELL would you believe it!

Firstly, the most incredible part: I think I managed to 'hack' my way through an exercise which could be VERY insightful here. You'll laugh when you see how simple what I 'achieved' was (if I did indeed achieve what I think I did!)

Secondly... I wanted to try doing the grep/count command on Mac, on matching folders from the mac export (before script run and converted).

I changed directory to do what we did on LInux, but soon learned (when Mac terminal reported it) that the .mbox files are actually directories. So the same command won't work, but I then went a level deeper into the .mbox folder, to see two files in each (mbox, table of contents).

I thought a count on the mbox would be the same as the counts we did on linux, though I confess I was just guessing my way through hoping my logic was't too far off.

I ran grep -c 'From ' 'mbox' once inside each of the three folders I counted yesterday on Fedora.

Well lookie here:

Trash Archive - 9948
Sent Archive - 5176
Keep Archive - 1624

The numbers I got doing the grep on Linux (on converted files in exptree folder) were:

Trash - 9948
Sent - 5176
Keep - 1627 (3 more on Linux there)

Have a I made as much progress as I think? (In confirming your suspicion of wonky Mac Mail msg counts?!

[cleidigh]

Yes, I think that is very clear here, good sleuthing.
Don't forget any small deltas are message From... Lines in bodies.
One thing would be to go to one of the folders in MM and see what the count is when you select all.
Regardless, The mbox files exported are being imported with whatever is in them.
Christopher

[johnstonesnow]

Yes I actually tried that yesterday (select all mails). It doesn't show how many are selected, I am sure it did years ago.
My current plan (still clearing space) is to force a full reindex of mail on Mac. Then I will see if the numbers change. Hopeful and expectant that they will. If so, I will feel 100% sure we are good.

I have trouble with OCD, so tell me, is this my OCD or does it make ANY sense doing (as I'd like to)...

After reindex, actually regardless of whether numbers change or not in mac Mail, I fancy doing it all again, export mac mail (after reindex), reconvert, reimport to TB on Linux.

Am I mad wanting to do that? Or would you agree it would be worthwhile?

thank you

[cleidigh]

No not OCD. The massive delta in the counts presumably is because of bad housekeeping in MM. Something like not reseting a counter after a big delete. If you really need more you could check the actual mbox files used in the system as opposed to the export.
Christopher

[johnstonesnow]

I like your thinking! I'll have a go at that.
Overnight I will run a reindex, and tomorrow I expect to see better numbers but will check live mbox files too. One way or another I think I will suss it now, will let you know the outcome, and I'll be visiting the pluginpage as mentioned earlier, over next few days when I get a minute free to express my gratitude.
Take care for now Christoper and thanks again.

[cleidigh]

BTW, I would also keep your original export just in case the rebuild does something funky.

[cleidigh]

sounds good, but hopefully you can do something better with your weekend %-}

[johnstonesnow]

Yes, remember what I said about OCD :-D

By the way, I just found the folder containing all mail data for Mac Mail. It claims to contain 31.67GB of data. That's worrying.

I guess it's possible that Mac Mail screwed up its own export, meanwhile your script converted everything it was asked to, and that's where the discrepancy lies.

Reindex first. Then export again. I may also try taking the live maildata folder to the better iMac, create a temp user to push that data into, then try exporting from mac mail there, would help me rule out this old dog of a Mac faiing on export.

"If you really need more you could check the actual mbox files used in the system as opposed to the export." - I just tried that. when i go to the directory in terminal, even just 'ls' command fails with "Operation not permitted". Maybe it's a 'mac thing' that you can't dig in that much. MM is not running so can't see why I am not allowed. But more to the point, I looked in Finder and of course prior to export it's all emlx files in a huge nested tree of folders.

Will just go with a reindex and hope things become clearer. Thanks again C

PS Ha, what's a weekend?!

[cleidigh]

far, far away in an unknown galaxy and planet, there was a place where people actually valued there time and actually named a part of their week, just for pleasure and "chilling"

...Never heard of it.

[johnstonesnow]

hahaha. you and me both.

[johnstonesnow]

Well the reindex wouldn't work but I eventually found out it works better in Safe Mode. Did that and successfully reindexed all Mail data on Mac. Then I ran a new full export of all local folders, just as last time. But last time the total size of export was 9.81GB. This time it's 10.27, that's a fair bit more, and i haven't added any emails to those folders, so maybe the reindex helped. Soon find out, going to try the conversion script again overnight and see what it looks like when i import the new set to Thunderbird. Will post the result

[cleidigh]

That delta is odd.
Will be interesting to see exports.
Christopher

[johnstonesnow]

If short on time please ignore this post, it's just an update but it's irrelevant now as my next post will explain.

It's all very odd! Here's the latest...

I did as i said above, did a fresh python conversion, and a fresh import into a new folder created for the purpose in TB.
This time, again all the smaller folders have identical numbers to MM. But the few big ones we've had trouble with, whilst STILL HAVING LESS than MM, had several thousand more this time!

Example, Sent Items Archive:
MM - 8861
TB after our first attempt said 5116
This time round: 8855

So this second batch seems to have fixed that folder and got them all this time.

I have a Trash Archive I am still struggling with. 13925 on MM. 9672 on TB first time, exactly the same second time.
I am now running a new import JUST of that one folder, to see if more 'make it' into TB. Assuming they are in the file, I think they are. (Edit: This still only brought in 9673, not 13k+ but I wonder if it may just be TB bugging out a bit, for example between conversations/msgs. Either that or the script doesn't like folders with more than 10k msgs inside.

So currently I am wondering if it's just too much to ask of the script, these huge folders with over 10k msgs inside.

Truth is, after refreshing my head and taking a renewed look at the situation, all the most important stuff looks to be in TB. Those old archive folders are just there for future legal reasons, so they are not unimportant, but I don't actually NEED them on my fedora laptop. So I am thinking worst case I just do this:

Store MM export AND the converted exptree on a USB stick in firesafe. I can import (either to a mac or TB) in the unilkely event I need to trawl back through those old mails any time in future.

[johnstonesnow]

Ok, I think I have everything sorted out.

Won't bore you with the details but I did lots of manual checks and can't find any missing emails, it's possible there are, but I took backups to put in the firesafe (of MM and TB) so I do 'have' it all if ever need to find a very old email in the future.

I used duplicate remover addon to clean up and it did find a good few thousand dupes. I still suspect Mac email counts could have been 'wonky' as you first suspected. Either way, I have everything secured (if not on this machine) and so I now have a good working Thunderbird installation with all my important emails in the folders they should be in :-)

Thank you again Christopher, you were beyond superb here. I have made a donation (through a friend with Paypal) and will write a review on the plugin page shortly. Take care of yourself and keep up the amazing work you're doing.

[cleidigh]

Mark
Thank you so much for your kind contribution. However, your review is really incredibly heartfelt. As I indicated before it's simply a pleasure working with someone who is patient and understanding that these types of issues have multiple variables that may or may not involve IETNG or Thunderbird.

I have enjoyed working together on this and of course am here if you run into anything else.

Follow up:

For storage, probably archival quality optical discs then high quality HDD. Definitely not solid state devices.

I can get you the more complex grep regex that distinguishes between message From separators and From in bodies. When you get counts like you do it should be obvious.

On the script, it's actually not written by me, but @obar who really deserves a lot of credit (thanks @obar!)
It's also important to understand the scipt does not modify the mbox files, it just puts them in a different directory.

Neither IETNG or Thunderbird have issues with file sizes as yours. I have users with 7-12GBs

I think all but the Trash is on target and I just think it's a mac mail thing. We proved whatever is being exported is being imported.

Finally, isn't it also possible to export to individual eml files? That would allow another comparison.
Christopher

[johnstonesnow]

Appreciate your comments fully Christoper, but as a recipient of such great help (rare in my experience for 'noobs' like me), and with you being the giver of such help, I argue strongly that I am the most pleased beneficiary of the interaction. :P

"Definitely not solid state devices." - Wow, glad I asked! I'd have thought they were the most 'stable' due to least moving parts, but your reply shows why I felt it best to check my 'assumptions', those being the mother of all mistakes of course!

Thank you to @obar from me too!

Re eml files - when I get some time I may have a look into that. For now, I feel I can finally have a bit of confidence in this machine and can finally STOP using the imac I have long wanted to use for target practice :). (Backups will be in safe in both folder structures so i should be good i think regardless of what weirdness was going on!)

[cleidigh]

Excellent.
The reasons against SSDs are that they use NAND gates which have a finite write/erase cycles and can have breakdown over extended times. HDDs do have mechanical aspects, however, in a non powered scenario, the media approaches tape drives, the gold standard.
C

[johnstonesnow]

Thanks!
I see, very interesting indeed.
25-30 years ago I worked in a datacentre loading large magnetic tapes all day and night (longest shift was 36 hrs straight, nearly rolled car on way home, never did a 36 hr stint again!)
So a decent quality HDD is about as good as I can get with what I have here (and can't buy gear just for this). I will format a hard drive, look for a way to encrypt the file, and get it stored. thanks!

[cleidigh]

p.s.
Have one in the Red Keep
Tyrion

[johnstonesnow]

Yes, it did take me a minute or so. I'm English after all. :-P