Windows users with non-ASCII in filepath can't load LuaGlobal

[Kebap]

Brief summary of issue / Description of requested feature:

1. Mudlet re-installed and started with new profile.
2. Error message appears, see below.
3. Mudlet is missing core functionalities, for example Geyser.

Special characters in user / directory name should not influence / hinder Mudlet functionality.
This seems to have come in a somewhat recent update to Mudlet.
Also breaks much more functionality than merely Geyser.

Steps to reproduce the issue / Reasons for adding feature:

1. Could be reproduced multiple times on my computer Profile keyboard navigation #1 but not on my computer Profile keyboard navigation #2
2. Full path is C:\Users\Eingeschränkt\AppData\Local\Mudlet - maybe is because of the special characters in user name?
3. @keneanung commented: When we implemented the auto-updater, we changed the installation directory from C:\Program Files\ to the user profile and I can't remember that we changed anything in that place lately. The code looks as if it should handle UTF-8 pathnames correctly

Error output / Expected result of feature

[ERROR] LuaGlobal.lua compile error
Error from Lua: cannot open /LuaGlobal.lua: No such file or directory

Extra information, such as Mudlet version, operating system and ideas for how to solve / implement:

Mudlet 3.8.1 on Win7

[SlySven]

Windows is known to be odd with path names - and the file functions in Lua do have to be fed with the correctly slashed pathnames. In the C++ code if we do not use C++11 raw string literals (and we may be forced not to to handle a Qt bug with QObject::tr(...)) a single / in a hard coded path name for *nix systems has to be doubly-escaped to \\\\ if it is going to be sent to a lua function - the C++ compilation strips each \\ down to \ and the same thing happens in the lua interpreter.

As it happens that path in the [ ERROR ] message looks as though the path to prepend to the LuaGlobal.lua filename was left as a ./ default so that it was expected to be in the same directory as the Mudlet executable - however for *Doze that at least should have been changed to .\\ or rather .\\\\ in the C++ source code!

[SlySven]

This also means I think that the QDir::nativeSeparators(...) static method does NOT do the right thing if we are writing a path/file name as a raw string in the C++ code that is going to be fed into the Lua interpreter as a string.

[Kebap]

Not sure I understand your comments or whether they were actually even directed towards me.. :)

Do you see any chance of me fixing this issue, short of configuring a whole new user from scratch?

[SlySven]

It is more of a general observation for anyone taking a look at it. It ought to be possible to fix it by looking at the path(s) used to get the LuaGlobel.lua file loaded, specifically on the Windows platform. I have a suspicion that we have left that OS use some default value - which may be "the same directory as the executable" but which is not suitable because of the non-POSIX path.

Looking at what I think is the cause of this problem:

void TLuaInterpreter::loadGlobal()
{
#if defined(Q_OS_MACOS)
    // Load relatively to MacOS inside Resources when we're in a .app bundle,
    // as mudlet-lua always gets copied in by the build script into the bundle
    QString path = QCoreApplication::applicationDirPath() + "/../Resources/mudlet-lua/lua/LuaGlobal.lua";
#else
    // Additional "../src/" allows location of lua code when object code is in a
    // directory alongside src directory as occurs using Qt Creator "Shadow Builds"
    QString path = "../src/mudlet-lua/lua/LuaGlobal.lua"; // <== A
#endif

    int error = luaL_dofile(pGlobalLua, path.toUtf8().constData());
    if (error != 0) {
        // For the installer we do not go down a level to search for this. So
        // we check again for the user case of a windows install.

        // overload previous behaviour to check by absolute path as well
        // TODO this sould be cleaned up and refactored to just use an array and a for loop
        path = QCoreApplication::applicationDirPath() + "/mudlet-lua/lua/LuaGlobal.lua"; // <== B
        if (!QFileInfo::exists(path)) {
            path = "mudlet-lua/lua/LuaGlobal.lua"; // <== C
        }
        error = luaL_dofile(pGlobalLua, path.toUtf8().constData());
        if (error == 0) {
            mpHost->postMessage("[  OK  ]  - Mudlet-lua API & Geyser Layout manager loaded.");
            return;
        }
    } else {
        mpHost->postMessage("[  OK  ]  - Mudlet-lua API & Geyser Layout manager loaded.");
        return;
    }

    // Finally try loading from LUA_DEFAULT_PATH
    path = LUA_DEFAULT_PATH "/LuaGlobal.lua"; // <== D
    error = luaL_dofile(pGlobalLua, path.toUtf8().constData());
    if (error != 0) {
        string e = "no error message available from Lua";
        if (lua_isstring(pGlobalLua, -1)) {
            e = "[ ERROR ] - LuaGlobal.lua compile error - please report!\n"
                "Error from Lua: ";
            e += lua_tostring(pGlobalLua, -1);
        }
        mpHost->postMessage(e.c_str());
    } else {
        mpHost->postMessage("[  OK  ]  - Mudlet-lua API & Geyser Layout manager loaded.");
        return;
    }
}

This immediately seems a bit dodgy because A to D are all wrong for Windows, for instance A should be "..\\\\src\\\\mudlet-lua\\\\lua\\\\LuaGlobal.lua" but the others need run-time replacements of / to \\\\ in both the literal strings AND the variables that are included. The original error at the top of the topic is because LUA_DEFAULT_PATH is empty at the time it is used on Windows!

Remember if debugging this the on-screen [ ERROR ] message will reproduce a path that has gone through both I think \\ to \ un-escapings so should look like a proper, real Windows path on-screen - which would be \LuaGobal.lua in the case given - which is probably also wrong and should have been .\LuaGobal.lua perhaps - so LUA_DEFAULT_PATH ought to have been .\\\\ instead?

[vadi2]

Lua on Windows handles / as a directory separator just fine though.

[SlySven]

That hasn't been my somewhat limited experience - IIRC there is a configuration setting somewhere in the lua stuff that contains a 4 character array which holds the compiled in settings for, amongst other thing, the wildcard character in package names and the directory separator. I recall this because when I fixed up the (internal to) LuaGlobal.lua handling of Windows vs. *nix file paths in the past I did use a check on, I think on one C array index into the config char array for '\\' or '/' - though I think a better solution was found subsequently.

Ah, ha - yes - see the package.config variable - that is the thing, from the Lua Unofficial FAQ:

1.40 Compatibility issues between Windows and Unix?

Here, 'Unix' stands for any POSIX-like operating system such as Linux, Mac OS X, Solaris, etc.

package.config is a string where the first 'character' is the directory separator; so package.config:sub(1,1) is either a slash or a backslash. As a general rule, try to use this when building paths.

A big difference between the Windows and Unix builds is that the default package.path is based on the location of the Windows executable, whereas on Unix it is based on /usr/local/share/lua/5.1. It is therefore easier to do a local user install of Lua on Windows, but Lua respects the environment variables LUA_PATH and LUA_CPATH.

Lua depends more directly on the system's C runtime libraries than most scripting languages so you have to appreciate platform differences. Use the "rb" specifier with io.open if you need compatibility with Windows binary I/O. Be careful of os.tmpname because it does not return a full path on Windows (prefix with the value of the TMP environment variable together with a backslash first.) os.clock is implemented very differently on Windows.

Likewise, os.time can actually crash Lua if passed non-compatible format specifiers. (This is no longer a problem with Lua 5.2, which does sanity checks first.)

For Windows GUI subsystem, os.execute can be irritating, and io.popen simply won't work - cross-platform extension libraries are available in this case.

[vadi2]

'as a general rule' - that does not counter the fact that / as a directory separator works fine on Windows in Lua.

[SlySven]

That is just it - for some, including myself, that does not work - it may well be that I do not have a lua installation prepared by following the normal quoted recipe (or that I have a Cygwin install around as well).

I wonder - are you confusing, by any chance, the lua subsystem's (or rather the package part of it) handling of directory separators - which can be configured either way (or possibly for something else entirely for say ' for RISCOS apparently!), the Windows cmd or powerline shells which will accept both or the Qt C++ core which will also handle both?

It does confuse the heck out of me as to what is supposed to work for a lua installation compiled on a Windows platform - ah, could it be that msys is POSIX-ish but mingw is Windowish?

[vadi2]

No I'm not, and it's also why using / in LuaGlobal.lua works for many people on Windows in its current state...

Path separators are not the problem here.

[SlySven]

So presumably that lua interpreter in the mingw from Qt is configured correctly for Windows, and it is using the same liblua that the Mudlet application gets linked to {it may not be, I suppose, for everyone}. For instance, luarocks provides a lua 5.1 interpreter which it can use if no other is found but by the same token itits libararies might get used instead.

💡 Ah, I wonder, @Kebap what do you get if you try to get the values of package.config on that setup that is giving you the error? For me at the command line, for instance, on my currently running system:

[stephen@ripley ~]$ lua51
Lua 5.1.5  Copyright (C) 1994-2012 Lua.org, PUC-Rio
> print(package.config)
/
;
?
!
-
> ^D
[stephen@ripley ~]$ 

Note the first value / which is the current directory separator that will be used when loading packages - like the LuaGlobal.lua file! Of course without the external lua scripts being run/available I cannot recall whether it is possible to run stuff from the "command line" in Mudlet. I'll try and fire up my 'Doze laptop machine later and see what I get on that one...

[Kebap]

Sure can try:

Seems like there is a backward-slash where you had a forward-slash

Not sure if I should try this in Mudlet though, as you seem to have used a different way and place..?

[Kebap]

I could create a new user or two, with and without special characters, just to make sure if the user name is really the issue here. Do you think it's helpful?

[vadi2]

Yes, please try

…

On Wed, 25 Apr 2018, 6:48 pm Kebap, ***@***.***> wrote: I could create a new user or two, with and without special characters, just to make sure if the user name is really the issue here. Do you think it's helpful? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1616 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAGxjN9JHbiPCOgUf3u1u8jVg0KjDiSbks5tsKj3gaJpZM4TZAbQ> .

[SlySven]

As far as I can see, the code is trying all the suggested locations for the LuaGlobal.lua files and finally reaches the one where it trys:

LUA_DEFAULT_PATH "/LuaGlobal.lua"

however from the resultant error message it seems that LUA_DEFAULT_PATH is an empty string so the used path is:

/LuaGlobal.lua

assuming (and I am still not convinced but that is hopefully just my problem) that '/' IS a workable directory separator where it is being used it means look for the file in the root of the file-system - on POSIX ones that is literally the root and on Windows it is the root on the currently active drive, probably C:\ (or C:/ 😉) - if we really want it to be the current working directory then LUA_DEFAULT_PATH should be a single . character and not completely empty - that being the case the error message would either return:

... cannot open ./LuaGlobal.lua ...

or if it returns the full path, possibly something like:

... cannot open C:/Users/Eingeschränkt/AppData/Local/Mudlet/LuaGlobal.lua ...

maybe? 😮

I see that the error message from the lua interpreter is captured in a std::string and that is sent to cTelnet::postMessage(...) via a std::string::c_str() conversion to a const char * but as the postMessage is expecting a QString this should engage the QString::QString(const char *str) constructor which uses the QString::fromUtf8() converter so non-ASCII characters should make it through undamaged... 😌

<aside> Continue the possibly off-topic bit about what the particular lua interpreter instance is using as it's directory separator I'm just trying to think where the run-lua-code-v4.xml file/package which provides the ^lua (.*)$ alias gets inserted into the profile - I think it is before/independent of the loading of LuaGlobal.lua - so you should be able to do:

lua print(package.config)

from the Mudlet profile command line even with the error you are getting.

This is a little speculative but as a runtime fix I wonder if it is possible to do something like:

lua package.config = "/;?!_"

to change the separator to / - although you may want to change the "command separator" setting in the preferences so that it does not split the above at the ; - and then see if you can "run" the LuaGlobal.lua file manually? Oh, that won't provide [ ERROR ] type error messages if it doesn't work though. </aside>

[Kebap]

Confirmed error with Mudlet 3.8.1 on Win 8.1 with user-name with special char: ä.
No error with user-name without special chars.

[Kebap]

lua print(package.config) without visible result
lua print('test') also without visible result
luaalias is available, but print does not seem to be
lua echo('test') works as expected

[Kebap]

Ran lua package.config = "/;?!_"
Error log says: ERROR:[string "Alias: run lua code"]:4: [string "package.config = "/"]:1: unfinished string near '<eof>'

Changed command seperator from ; to something else

Ran lua package.config = "/;?!_"
OK, I guess? Result invisible

Ran lua run("./LuaGlobal.lua")
Error log says: ERROR:[string "return run("./LuaGlobal.lua")"]:1: attempt to call global 'run' (a nil value)

Test with include - same error.

lua require("./LuaGlobal.lua") - interesting error:
ERROR:[string "return require("./LuaGlobal.lua")"]:1: module './LuaGlobal.lua' not found: no field package.preload['./LuaGlobal.lua'] no file 'C:\Users\Eingeschränkt\.config\mudlet\profiles\new profile name\\/LuaGlobal\lua.lua' no file 'C:\Users\Eingeschränkt\.config\mudlet\profiles\new profile name\\/LuaGlobal\lua\init.lua' no file '.\\/LuaGlobal\lua.lua' no file 'C:\Users\Eingeschr�nkt\AppData\Local\Mudlet\app-3.8.1\lua\\/LuaGlobal\lua.lua' no file 'C:\Users\Eingeschr�nkt\AppData\Local\Mudlet\app-3.8.1\lua\\/LuaGlobal\lua\init.lua' no file 'C:\Users\Eingeschr�nkt\AppData\Local\Mudlet\app-3.8.1\\/LuaGlobal\lua.lua' no file 'C:\Users\Eingeschr�nkt\AppData\Local\Mudlet\app-3.8.1\\/LuaGlobal\lua\init.lua' no file 'C:\Users\Eingeschränkt\.config\mudlet\profiles\new profile name\\/LuaGlobal\lua' no file '.\\/LuaGlobal\lua.dll' no file 'C:\Users\Eingeschr�nkt\AppData\Local\Mudlet\app-3.8.1\\/LuaGlobal\lua.dll' no file 'C:\Users\Eingeschr�nkt\AppData\Local\Mudlet\app-3.8.1\loadall.dll' no file 'C:\Users\Eingeschränkt\.config\mudlet\profiles\new profile name\' no file '.\.dll' no file 'C:\Users\Eingeschr�nkt\AppData\Local\Mudlet\app-3.8.1\.dll' no file 'C:\Users\Eingeschr�nkt\AppData\Local\Mudlet\app-3.8.1\loadall.dll'

Just to be sure lua require("/LuaGlobal.lua")
Error log says:
ERROR:[string "return require("/LuaGlobal.lua")"]:1: module '/LuaGlobal.lua' not found: no field package.preload['/LuaGlobal.lua'] no file 'C:\Users\Eingeschränkt\.config\mudlet\profiles\new profile name\/LuaGlobal\lua.lua' no file 'C:\Users\Eingeschränkt\.config\mudlet\profiles\new profile name\/LuaGlobal\lua\init.lua' no file '.\/LuaGlobal\lua.lua' no file 'C:\Users\Eingeschr�nkt\AppData\Local\Mudlet\app-3.8.1\lua\/LuaGlobal\lua.lua' no file 'C:\Users\Eingeschr�nkt\AppData\Local\Mudlet\app-3.8.1\lua\/LuaGlobal\lua\init.lua' no file 'C:\Users\Eingeschr�nkt\AppData\Local\Mudlet\app-3.8.1\/LuaGlobal\lua.lua' no file 'C:\Users\Eingeschr�nkt\AppData\Local\Mudlet\app-3.8.1\/LuaGlobal\lua\init.lua' no file 'C:\Users\Eingeschränkt\.config\mudlet\profiles\new profile name\/LuaGlobal\lua' no file '.\/LuaGlobal\lua.dll' no file 'C:\Users\Eingeschr�nkt\AppData\Local\Mudlet\app-3.8.1\/LuaGlobal\lua.dll' no file 'C:\Users\Eingeschr�nkt\AppData\Local\Mudlet\app-3.8.1\loadall.dll' no file 'C:\Users\Eingeschränkt\.config\mudlet\profiles\new profile name\/LuaGlobal' no file '.\/LuaGlobal.dll' no file 'C:\Users\Eingeschr�nkt\AppData\Local\Mudlet\app-3.8.1\/LuaGlobal.dll' no file 'C:\Users\Eingeschr�nkt\AppData\Local\Mudlet\app-3.8.1\loadall.dll'

[SlySven]

Strange about print not working - I though it was a lua built-in - but:

lua echo(package.config)

should work as well.

Looking at the output from the first sort of working example the error output is trying to load the LuaGlobal.lua file - lets see what file names and paths are being tried:

1. C:\Users\Eingeschränkt\.config\mudlet\profiles\new profile name\\/LuaGlobal\lua.lua
2. C:\Users\Eingeschränkt\.config\mudlet\profiles\new profile name\\/LuaGlobal\lua\init.lua
3. .\\/LuaGlobal\lua.lua
4. C:\Users\Eingeschr�nkt\AppData\Local\Mudlet\app-3.8.1\lua\\/LuaGlobal\lua.lua
5. C:\Users\Eingeschr�nkt\AppData\Local\Mudlet\app-3.8.1\lua\\/LuaGlobal\lua\init.lua
6. C:\Users\Eingeschr�nkt\AppData\Local\Mudlet\app-3.8.1\\/LuaGlobal\lua.lua
7. C:\Users\Eingeschr�nkt\AppData\Local\Mudlet\app-3.8.1\\/LuaGlobal\lua\init.lua
8. C:\Users\Eingeschränkt\.config\mudlet\profiles\new profile name\\/LuaGlobal\lua
9. .\\/LuaGlobal\lua.dll
10. C:\Users\Eingeschr�nkt\AppData\Local\Mudlet\app-3.8.1\\/LuaGlobal\lua.dll
11. C:\Users\Eingeschr�nkt\AppData\Local\Mudlet\app-3.8.1\loadall.dll
12. C:\Users\Eingeschränkt\.config\mudlet\profiles\new profile name\
13. .\.dll
14. C:\Users\Eingeschr�nkt\AppData\Local\Mudlet\app-3.8.1\.dll
15. C:\Users\Eingeschr�nkt\AppData\Local\Mudlet\app-3.8.1\loadall.dll

I wasn't sure where the file should be located these days so I downloaded and installed the 3.8.1 binary and found that the path used is:

C:\Users\Stephen\AppData\Local\Mudlet\app-3.8.1\mudlet-lua\lua

also with that Windows Installer version, I have the following for the package.config:

lua print(package.config)
\
;
?
!
-

so paths to it must use \ for the package handler to work IMO. It is still not clear though why different people are getting different things.

Edited: to quote the list of paths with ` rather than '

[SlySven]

Why some paths are mojibake and others correctly display the Username must be because they are coming from different sources and some of them are not properly capable of I18n yet - it would be helpful if we can track those down and fix them by Mudlet 4.0 - but they may not be ones we have control over. Note that it cannot be the display code or fonts because the individual error messages all pass through the same process one they are created.

[SlySven]

Just to be sure that I was using the expected instance (as a developer I have more than one copy on that PC) I renamed the file and got the same result:

Also I created a new user WITH non-ASCII (and probably not even Latin1/ISO 885901) characters and I can confirm that things are messed up in the same way for that user - Mudlet cannot see the file in:

C:\Users\şțȅƥĥēƞ\AppData\Local\Mudlet\app-3.8.1\mudlet-lua\lua\LuaGlobal.lua

but it can for

C:\Users\stephen\AppData\Local\Mudlet\app-3.8.1\mudlet-lua\lua\LuaGlobal.lua

😭

[SlySven]

This Q&A on Stack Exchange does not sound promising.

However the MIT licenced luawinfile Windows (only compile on mingw NOT MSVC) filename translator (converts UTF-8 file names to/from the UTF-16 that must be used with the Windows file-handling API) looks a bit better - it provides replacements for the LFS module that can take UTF-8 file and path names.

[vadi2]

Good find. Just to throw it out there, we have https://github.com/starwing/luautf8 already in Mudlet, is there anything in that which could help us?

[Kebap]

SlySven, your list of 15 directory names seems wrong. For example, 13 should read .\.dll and not ..dll

[SlySven]

Ah, that is the mark-up system interfering - in some contexts in here a backslash escapes the following character which is needed e.g. if you want to show a less than symbol in ordinary text like this "<" you will not be seeing the backslash there... and I'd used single ordinary quotes rather than the code quotes - edited to fix the quoting.

[SlySven]

The luautf8 is all about UTF-8 string handling - I do not think it will help here as luainfile is specifically about interfacing with the Windows file handling (which only works with UTF-16 strings) C or more likely C++ library - it looks as though it needs to either wrap around or replace lfs on Windows only (not Cygwin though, I'd bet)...

[vadi2]

Hmm but we do not use lfs to load LuaGlobal.lua, so how will luainfile solve the problem?

[keneanung]

It's not only a replacement for lfs. It also has a replacement for dofile and loadfile. We'll probably need to replace our call to luaL_dofile with a lua call to dofile.

Edit to add: The luainfile library links (by default) against lua 5.3... Not sure if that is a hard dependency or not.

[vadi2]

Seems like we can just write our own luaL_dofile function then that works with Windows' special encoding?

[Kebap]

How can we push this forward?

[SlySven]

I have raised an issue on the luawinfile repository asking about 5.1 compatibility but I have taken a look and it does actually use some 5.3 language features that are not provided by lua 5.1 - there is a separate, official, 5.3 compatibility module that attempts to provide some of the missing features to 5.2 and 5.1 but it is not without some other complications so is not a drop in solution.

[Kebap]

Seems related to #229

[Kebap]

How can we push this forward?

[SlySven]

We will need to backport the MIT licensed https://github.com/cloudwu/luawinfile (a single C file of around 884 lines {808 soc} of code) from 5.3 to 5.1 to accommodate some features that are not present in the earlier version of lua that Mudlet uses. This will require someone with a better grasp of lua-C coding than I feel up to currently, I think. However it seems to be something that can be done and tested independently without worrying too much about the rest of the code-base - though it can only really be tested on a Windows development platform.

[vadi2]

The other issue is that people can't save their profiles on windows with non-english usernames - no history is written. Don't think we have a ticket for that.

[Kebap]

@SlySven
Did you maybe learn some things during your recent fixes for bringing international letters to Mudlet, to bring this issue forward?

[SlySven]

Not really, I do think that back porting luawinfile to Lua 5.1 offers our best hope but I just do not have a good enough grasp (yet?) of the C/C+++lua API to place any confidence in my ability to do this. Does anyone else? I guess we need to look through the code for the official Lua io library and see how luawinfile replaces some (all, I don't know?) of the functions with Windows specific replacements.

As people may have noticed my primary Windows development platform is not the most conformant (a Windows 7 laptop with a 64-bit processor but running the 32-bit version) - my main PC also has a 64-bit Windows 7 but that triple-boot machine has not run that OS for months so I have several hours of staring at a "Updating Windows... do not switch off" screen to look forward to and I am not that motivated, ATM!

Although, now that the Qt on-line installer finally seems to have switched to offering only a 64-bit Mingw64 Windows install - compared to the previous 32-bit only Mingw one I might considered doing so if I have some hours to throw away...

[vadi2]

Looking good...

This is thanks to https://gist.github.com/Egor-Skriptunoff/2458547aa3b9210a8b5f686ac08ecbf0 which was created at the beginning of the year.

[vadi2]

Will be available in Mudlet 3.22