Rust or Cython to speed up core components

[jnewb1]

Has any thought or development gone into using cython for this library, especially the core components which could benefit from a speedup (move generation and validation)? I think that might be a good way to speed up the library and any chess engines that use this library.

I tried just compiling just the core components using cythonize, but main issue right now appears to be lack of support for Generic and TypeVar from cython's end. (see cython/cython#2753)

I think if we get a basic cython build running and show the performance delta, that might be a good start (I'm interested in working on that, I was curious if anyone else had done something similar yet).

[niklasf]

I previously rejected ideas diverging from pure Python to retain more flexibility, but now that we're on version 1.x it might be time to reconsider.

Other similar ideas were to use C/C++/Rust for some performance critical paths.

[jnewb1]

Awesome! Maybe I'll try to get a cython build of the move generation working to see what kind of performance increases we could get. I do agree that pure python is much cleaner, but the speed improvements could be substantial, A small C/C++ library would also do nicely, or perhaps we could use an existing chess library and just add an option to use that library rather than the python core components. If anyone else is interested in this or has other ideas, please leave them here.

[jkterry1]

Wouldn't C/C++/Rust make more sense than Cython?

[jnewb1]

Well the idea with cython is that most of your python code remains the same, but yea I think a C/C++ helper library would also make sense

[sudochop]

+1 If others are like me, this is a huge want as I use this library for processing large pgn files and my only solution left for speedups is renting beefier vms.

[niklasf]

Potentially usefu for this: A C++ translation of the core module: https://github.com/pjpuzzler/cpp-chess (thanks for sharing @pjpuzzler!)

[pawngrubber]

@niklasf I don't know enough about this repository to know where the bottlenecks are, but I know enough about Cython to take a swing at writing a PR. Can you point me in the right direction for identifying bottlenecks? For now I'm mostly interested in faster PGN reading and move generation (for example speeding up perft). Also, if I want to write any tests for my PR where should I start?

[niklasf]

@pawngrubber Maybe profiling python examples/perft/perft.py examples/perft/random.perft could show some points of interest. Other than that I don't really know what to do, except trial and error.

python test.py is a fairly comprehensive test suite, so we might not need much more (for existing functionality).

[tissatussa]

lately i stumbled upon "Python C extensions" at https://book.pythontips.com/en/latest/python_c_extension.html .. 3 methods are described to directly use C code in Python, without compiling the C code into a binary.

[tissatussa]

i guess, when writing a perft function with python-chess, the speed is most important .. my intuition tells me C is better than Python anyway, so why not leave such task to C and "embed" it.

[luki139]

I've been developing a Rust port (almost 1:1) of this library some time ago: https://github.com/lukaszpancer/RustChess
It implements only standard chess functionality, pgn parsing, syzygy endgame, and some basic Engine communication.
Right now i'm working on making this a python library with pyo3 (which is going quite well). And this python library is approximately 10x faster.

It might be a bit messy but it might help someone. I hope that authors don't mind.

[qwertyquerty]

Update on this?