Fix package exporter work for paths w/ international letters

[vadi2]

Brief overview of PR changes/additions

Try @SlySven's suggestion from #1832 (comment) and use the ZIP_FL_ENC_UTF_8 flag when giving the zip a name.

Motivation for adding to Mudlet

Fix package exporter to work for Windows locations with international letters in them (#1790 (comment))

Other info (issues closed, discussion etc)

Fixes #1993

[Kebap]

Using the "package exporter (experimental)" from menu, and selecting a few items to export (no additional files added), this still produces an empty 0kb XML file in the ZIP. :(

edit: win7 with non-ASCII filepath

[vadi2]

Try the new commit, though I am not certain it will address the issue.

[Kebap]

New commit does also fail.
WIn10 with non-ASCII user Name.

[SlySven]

This if can be removed if line 275 is changed to:

    QStringList contents = dir.entryList(QDir::NoDotAndDotDot);

However since the code here will die horribly if any directories are fed to zip_file_add it may perhaps be wise to actually use:

    QStringList contents = dir.entryList(QDir::Files);

at least until the code is fixed to handle modules with sub-directories...

[vadi2]

Yes, though the original problem is still there, so it doesn't really matter to move some code around... 😞

[vadi2]

I'll start looking into this again. I can just create a folder called тест in Windows to replicate the issue, right?

[Kebap]

I think I tested with a non-ASCII user name (windows user) and not quite sure if your way would yield the same result.

[vadi2]

How do you create such a user in windows?

[Kebap]

https://support.microsoft.com/en-us/help/4026923/windows-10-create-a-local-user-or-administrator-account

[vadi2]

First stumbling block: https://bugreports.qt.io/browse/QTBUG-70751

[vadi2]

Try the latest build, I've found the issue.

Mudlet didn't seem to want to install the package on the non-ASCII user for me, but that's separate to the package exporter not working.

[vadi2]

@Kebap ping

[Kebap]

Exporting a package with non-ascii name will now produce output just like with a only-ascii name, however that output seems corrupt. See screenshot of test.xml file which does contain weird characters at the start, and not only readable text. Maybe it is just my text editor, but this seems wrong. Attached you can find the test.zip file with text.xml inside. Again, same contents as if the zip is created with non-ascii name, so that is working (somehow)

[vadi2]

@Kebap is the exporter now fixed by chance, given @SlySven's fix that was merged in?

[SlySven]

Actually that text looks as though it is a sort of Pascal string formatted UTF-16 text (Pascal format means the characters of the text are preceded by an integer - 32 bit in this case - number of characters - which in this case seems to be the QChar UTF-16 units that Qt uses for text). It certainly isn't an ISO 8859-* or the UTF-8 encoding. From what I recall from poking around with the binary format details for saving and loading the map files, that is what you get when using QDataStream to write QStrings to/from a file.

<aside>Blast it, I have just spent a day upgrading my FreeBSD from 11.2-RELEASE to 12.0-RELEASE and it still includes a version 27.x PaleMoon web-browser (fork of Firefox which maintains the older more functional and less Chromelike add-ons API) whereas I need the 28.x one to get Github post previews and some other things, like Emojis, working 😢)</aside>

[Kebap]

Vadi asked me to check if the error persists in the latest development branch, but I did not come around to it yet. That is, not this PR, but the branch named "development"

[SlySven]

Sorry, I'm not following you - this PR only fixes the handling of files to put into the zip file which had names that our code could not convey into the zip library code because we were confining ourselves to a Latin1 encoding in line 289 as it is in this PR.

There are other changes which I guess have already made it into the package exporter which someone (@vadi2) will need to fixup to merge this PR into the main line code but provided you can store oddly named files in the package that is ALL that this PR addresses - so the fact that THIS PR seems to produce corrupt output is not the issue.

You did say it yourself:

Exporting a package with non-ascii name will now produce output just like with a only-ascii name,...

but as you then mentioned:

...however that output seems corrupt.

is that corruption ONLY for the non-ascii named file or is it the same for the same content in both files?

[Kebap]

Sorry for any confusion SlySven :) Like I said above, I am not going to test this PR, but only the main development branch compared to the current release version 3.16.1 - This may indeed be confusing, as we are writing in this PR. However, this is still the best place so far, as many issues are affected and no central issue links them all together like this PR does. So this is the update.

Following tests happened on Win10. International character tested was like äöü. Did not test Asian characters or alike.

There are quite a few racing error sources here, i.e. different places where the non-ASCII letters may occur, so seperate tests seem necessary as follows:

 

1. When starting the package exporter, user will be asked for a name. If non-ASCII is given, will export work?

Setup:

• ASCII Username in Windows
• ASCII Path to Package
• ASCII Profile in Mudlet
• Non-ASCII Package name entered

Test results:

• current release : Package is exported fine seemingly, the zip file has contents with non-ASCII packagename.xml and config.lua, as expected. But the XML has a size of zero!
• latest development: Package & contents exported fine.
• this PR: not tested.

 

2. When a user creates a new profile, they can put any name. Will non-ASCII characters affect exporting from that profile?

Setup:

• ASCII Username in Windows
• ASCII Path to Package
• Non-ASCII Profile in Mudlet
• ASCII Package name entered

Test results:

• current release: Does not allow creation of profile with Non-ASCII names
• latest development: Does not allow creation of profile with Non-ASCII names
• this PR: not tested.

 

3. When choosing which folder to export package to, user may choose a path containing a folder with non-ASCII in there somewhere, as described in Mudlet Crashing When Using Package Exporter #1790 (comment)

Setup:

• ASCII Username in Windows
• Non-ASCII Path to Package
• ASCII Profile in Mudlet
• ASCII Package name entered

Test results:

• current release: Package is exported fine.
• latest development: Package is exported fine.
• this PR: not tested.

 

4. If you install Mudlet as a non-ASCII user, Mudlet itself will live in a non-ASCII path. This destroys most Mudlet functionality and is discussed in Windows users with non-ASCII in filepath can't load LuaGlobal #1616 in more detail.

Setup:

• Non-ASCII Username in Windows
• ASCII Path to Package
• ASCII Profile in Mudlet
• ASCII Package name entered

Test results:

• current release: Package is exported fine seemingly, the zip file has contents with non-ASCII packagename.xml and config.lua, as expected. But the XML has a size of zero!
• latest development: Package & contents exported fine.
• this PR: not tested.

 

Summary:

• Test 1 and 4 - development version solves problems in current release
• Test 2 and 3 - no problems in either version
• This PR now seems obsolete!

[vadi2]

Nice - so we can close this pr, then!