Windows Path Canonicalization

[rocky]

I have been looking into the Windows failures that have been more prevalent.

Apparently changes in improving $InputFileName have made things worse. No good deed goes unpunished.

One problem stems from Windows predilection for using backslash as a path separator. Python uses backslash as an as an escape character in front of some letters, like "t".

When os.path.join() is used we get path names like C:\\Users\\rocky\\Appdata\\Programs\\.... In Python tests we will see ...\\test\\.... Oddly the \t sometimes gets treated as a tab!

Python provides PureWindowsPath.as_posix() which will convert backslashes to forward slashes in file paths.

In my opinion this is more readable and less prone to error.

I suggest then we canonical path names to the POSIX equivalent.

Anyone know what WMA does on MS Windows?

Thoughts?

[mmatera]

@rocky, probably the way to handle this error is to skip it in Windows. Another possibility would be to use the pathlib module, which works better cross-platform.
In any case, another error occurred in doctest, which I still was not able to understand: since the refactor, there is a point where doctest just crashes. I was doing experiments in #1010, but I couldn't find the reason for the failure. Last Friday I downloaded a virtual machine image from https://developer.microsoft.com/en-us/windows/downloads/virtual-machines/ to do a test locally, but I didn't have the time to look at it yet. I hope to be able to do it over the week.

[rocky]

probably the way to handle this error is to skip it in Windows.

The issue of ugly path names is still around and people will use Mathics on Microsoft Windows.

Another possibility would be to use the pathlib module, which works better cross-platform.

While I like pathlib, that is not the problem and I don't think that will solve confusion. The problem is in how this gets represented both internally and back to the user.

Last Friday I downloaded a virtual machine image from https://developer.microsoft.com/en-us/windows/downloads/virtual-machines/ to do a test locally, but I didn't have the time to look at it yet. I hope to be able to do it over the week.

This kind of thing happens a lot. We are always working on the same thing when it would be better if we worked on different things. I now have a Windows setup that I can use to check things over. I will spend a week on that. After that is done, I will stop and move onto other things. In fact, the reason I haven't been working on Mathics3 much is because I need to focus on my BlackHat Asia talk in mid April. (Yes, I need that much time on this because I need to update software as well as slides and probably write a "whitepaper" as well.)

For next week, I would greatly appreciate it if we you would focus whatever time you have to get doctests working by chapter and section. And then remove the obstacle in the doc system which prevents these highly touted Mathics3 modules (that we would like to replace everything with) work more along the same way that builtin modules work. Thanks in advance.

[mmatera]

@rocky
I just had the chance to check the issue with the chapters/sections tests. As far as I can see, the -c "[chapter name]" option works in master, both if we run over a Mathics-core chapter, or if we do it over a Pymathics module (at least, it works with pymathics.graph).
The option -s also works for individual symbols (builtin and pymathics-defined symbols).

What is not available now is a way to run all the tests in a section inside a guide. I guess this would be useful to check parts of heavy modules like pymathics.natlang.
In case we want to add this functionality, I think that the hardest part would be to find a suitable option name, let's say something like -S (for section) to distinguish it from -s (symbol), and what to do for modules that do not have "subsections".

[rocky]

The problem was not that the options were broken, but we are not accessing these by chapter or section but instead iterating over everything. This is bad for incremental update and autoload.

I mentioned this somewhere I can't find right now when we were having a discussion about the goals of the doc system. I said that being developer friendly is maybe a little more than "nice to have" .

[rocky]

In case we want to add this functionality, I think that the hardest part would be to find a suitable option name, let's say something like -S (for section) to distinguish it from -s (symbol), and what to do for modules that do not have "subsections".

I don't think you are misunderstanding something here as well. The structure of Mathics3 modules should allow for the same kind of flexibility of builtins. Currently, under pymathics.graph we can't add another module like pymathics.graph.parametric and create sections under than purely because the doc system does not currently track that.

[rocky]

One other thing with respect to Windows testing that I am not working on right now so that too would be good to too. (But personally, I would prefer it if we could finish things started rather than jump around).

If we create a CircleCI setup for Microsoft Windows, then when things break we can ssh into the CI server and debug things that way.

CircleCI really is much better than github actions and it is also generally faster too. There can however be a big learning curve and it is constantly changing more rapidly than actions. But the changes CircleCI makes is all to improve speed flexibility and long-term maintainability.

[mmatera]

My interest in fixing the Windows workflow is to see if we have introduced a kind of a bug in the doctest refactor (and also, to have a full CI working). In any case, let's leave this for another time.

[mmatera]

This kind of thing happens a lot. We are always working on the same thing when it would be better if we worked on different things. I now have a Windows setup that I can use to check things over. I will spend a week on that. After that is done, I will stop and move onto other things. In fact, the reason I haven't been working on Mathics3 much is because I need to focus on my BlackHat Asia talk in mid April. (Yes, I need that much time on this because I need to update software as well as slides and probably write a "whitepaper" as well.)

I started looking at this just because a part of the CI got broken. But OK, I leave this for you when you have the time.

For next week, I would greatly appreciate it if we you would focus whatever time you have to get doctests working by chapter and section. And then remove the obstacle in the doc system which prevents these highly touted Mathics3 modules (that we would like to replace everything with) work more along the same way that builtin modules work. Thanks in advance.

OK, I will focus o on this then.

[rocky]

Some small follow ups. First, I see that PurePythonPath that used in the "Windows-tolerance" branch that I am currently making changes is from pathlib. So we are using that, just not right now for joining path elements which is one way it is typically used.

Second, I have been wondering exactly where that tab got inserted by mistake and was worried that we might have a bug in our scanner. We do not. We handle "\t" and "\\t" and "a\\t properly as far as I can tell. So there may be is a bug or subtle interaction inside test.helper.check_evaluation() which is improperly turning the second trailing \t into a tab when there is an additional \ before it.

[rocky]

My interest in fixing the Windows workflow is to see if we have introduced a kind of a bug in the doctest refactor (and also, to have a full CI working).

Looking at the Windows failures I am seeing a some flakiness and weaknesses in test.builtin.files_io.test_files tests. It is good that we got these out of doctests so that we have the full flexibility of Python to use to ensure preconditions needed and not leave a mess when a test fails. However we need to start making use of Python to do this kind of thing.

I have added comments to #908