-
Notifications
You must be signed in to change notification settings - Fork 3.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Python 3.12 Peg Parser Grammar has a small problem #3957
Comments
The grammar is scraped from the pegen grammar 3.12.1. If it doesn't parse, it's caused by one of the following:
Pegen does not take a description for the lexical structure. It's backwards captured from an implementation then placed into the Python documentation--something that is really bad practice. We have to read the source code for the Python3 compiler and find out which. |
Ok, thanks for the quick response. Was hoping it was a quick fix. I am not familiar with g4 files, mainly just a consumer. |
The lexical analysis documentation does not mention the generation of the TYPE_COMMENT token. The solution for the comment in the example (# type: ignore) could be that the lexer would recognize it as a plain COMMENT (or perhaps as a hidden TYPE_COMMENT). Until then, I temporarily set the TYPE_COMMENT tokens to hidden in my own repository. |
We might be able to define an Antlr4 "lexer mode" to work around a parser-state-dependent lex, but "lexer modes" are basically hacks for the real deal. It would be best to have a parser-state aware lexer option for Antlr5. @ericvergnaud |
So I generated new lexer/parser files this morning and verified using the modified file provided by @RobEin ; They worked great and got me around this issue with the mypy commented files. This resolves the issue for me now as I don't look at that construct in my visitor code Thanks very much for this. |
Pycharm and Python 3.12 in general accept lines in this format
However, the Python parser generated by the above burps on the '# type: ignore' comment. Says it is a syntax error. Not sure why
The text was updated successfully, but these errors were encountered: