Add tests for new python syntax

[vors]

Hello!

Here are tests that are legal python syntax (i.e. https://github.com/astral-sh/ruff parses it fine) but they are currently failing

thread 'extract_py_imports::tests::test_syntax_constructs' panicked at python_extractor/src/extract_py_imports.rs:162:68:
called `Result::unwrap()` on an `Err` value: invalid syntax. Got unexpected token '[' at line 7 column 30

I'm not sure that currently used parser is a good long-term choice to avoid that kind of problems.
Maybe we can switch to ruff parser altogether?

[eed3si9n]

I think we use RustPython's parser, but the dependency is pinned to this commit - RustPython/RustPython@b9ed63e, which is 1409 commits behind RustPython/RustPython@b9ed63e...main.

[johnynek]

it looks like ruff doesn't seem to have an external parser dependency:

https://github.com/astral-sh/ruff/blob/main/crates/ruff_python_parser/Cargo.toml

I wonder if we can depend on their parser since I think that is fairly well tested across a bunch of code bases.

[vors]

Another alternative would be to depend on the native CPython ast parser and dump it into json with something like https://pypi.org/project/ast2json/ then parse import in rust or perhaps just do this whole thing in python, but that would mean starting brining dependencies outside of rust ecosystem which is big design decision.

[vors]

FWIW I found more constructs that don't work, I think it was mostly around the new match syntax.

[johnynek]

this is legal?

[johnynek]

I merged main to pickup: #237

That should fix match issues, but your test surprises me that it is legal syntax.

[vors]

Thank you, yes the [*foo] is legal syntax but pyright for example also has troubles with it at the moment. Ruff doesn't (and python 3.11.8 doesn't)