gh-118263: Generalize path_t for C level optimizations

[nineteendo]

Benchmark:

ntpath.py

script

::generalize-path_t.bat
@echo off
echo splitroot && call main\python -m timeit -s "import os" "os.path.splitroot('//server/share/spam/eggs')" && call generalize-path_t\python -m timeit -s "import os" "os.path.splitroot('//server/share/spam/eggs')"
echo normpath  && call main\python -m timeit -s "import os" "os.path.normpath('//server/share/spam/eggs')"  && call generalize-path_t\python -m timeit -s "import os" "os.path.normpath('//server/share/spam/eggs')"

Note: no difference for ntpath.is*() & ntpath.exists().

splitroot
1000000 loops, best of 5: 296 nsec per loop # before
2000000 loops, best of 5: 193 nsec per loop # after
# -> 1.53x faster
normpath
1000000 loops, best of 5: 287 nsec per loop # before
2000000 loops, best of 5: 171 nsec per loop # after
# -> 1.68x faster

posixpath.py

script

# generalize-path_t.sh
echo splitroot && main/python.exe -m timeit -s "import os" "os.path.splitroot('/spam/eggs')" && generalize-path_t/python.exe -m timeit -s "import os" "os.path.splitroot('/spam/eggs')"
echo normpath  && main/python.exe -m timeit -s "import os" "os.path.normpath('/spam/eggs')"  && generalize-path_t/python.exe -m timeit -s "import os" "os.path.normpath('/spam/eggs')"

splitroot
1000000 loops, best of 5: 258 nsec per loop # before 
2000000 loops, best of 5: 125 nsec per loop # after
# -> 2.06x faster
normpath
2000000 loops, best of 5: 198 nsec per loop # before
2000000 loops, best of 5: 103 nsec per loop # after
# -> 1.92x faster

• Issue: Generalize path_t for C level optimizations #118263

[nineteendo]

@barneygale, @ronaldoussoren could you review this pull request instead then?

[nineteendo]

@eryksun, I wasn't able to use this for _path_is*(), since I can't prevent a UnicodeDecodeError from being raised (like for b'\xff').
See https://github.com/python/cpython/actions/runs/8862420350/job/24335361259#step:5:709

I was able to speed up os.path.splitroot() & os.path.normpath() though.

[nineteendo]

I think you'll be particularly pleased with this addition (I don't know why we didn't do this earlier):

cpython/Modules/posixmodule.c

Lines 5564 to 5569 in 19209c6

	if (!norm_len) {
	result = PyUnicode_FromOrdinal('.');
	}
	else {
	result = PyUnicode_FromWideChar(norm_path, norm_len);
	}

[eryksun]

I wasn't able to use this for _path_is*(), since I can't prevent a UnicodeDecodeError from being raised (like for b'\xff').

The converter could set a flag in the path_t record to indicate that conversion failed with a ValueError, and another to indicate that the path has embedded null characters. There could be a single conversion option to repress raising ValueError that applies to both cases.

could we try to get this in 3.13? We only have 4 days left.

This change shouldn't affect anything compatibility-wise, and if so that needs to be addressed. It's support code for CPython. I think it could be added to beta 2 this month if it misses beta 1.

[nineteendo]

The converter could set a flag in the path_t record to indicate that conversion failed with a ValueError, and another to indicate that the path has embedded null characters. There could be a single conversion option to repress raising ValueError that applies to both cases.

I added a suppress option, which tells argument clinic you would like to handle errors manually, otherwise path_converter() will get too complex.

[eryksun]

I get a similar performance increase on Windows:

splitroot() -- 1.67x speedup

> .\python_main -m timeit -s "import os" "os.path.splitroot('//server/share/spam/eggs')"
1000000 loops, best of 5: 240 nsec per loop
> .\python_work -m timeit -s "import os" "os.path.splitroot('//server/share/spam/eggs')"
2000000 loops, best of 5: 144 nsec per loop

normpath() -- 1.56x speedup

> .\python_main -m timeit -s "import os" "os.path.normpath('//server/share/spam/eggs')"
1000000 loops, best of 5: 218 nsec per loop
> .\python_work -m timeit -s "import os" "os.path.normpath('//server/share/spam/eggs')"
2000000 loops, best of 5: 140 nsec per loop

There's no significant change in the performance of the exists() and is*() checks on Windows. I expected that since in those cases this PR just simplifies the implementation by removing the manual path_converter() call.

[nineteendo]

Would you accept this now?

[eryksun]

I requested @zooba and @barneygale to review your changes. Steve implemented the original changes to path_t to make it always use wide on Windows. Barney wrote the docstrings that you're simplifying to use a generic description.