Great refactoring & version 1.0

[xnuinside]

Dear community,

I am excited to share that my open source project is in need of support from the community. I have been receiving regular feature requests and I am grateful for the enthusiasm and interest in my project. I would like to encourage you to continue sharing your ideas and feedback.

However, I am currently facing a major challenge in terms of code refactoring. As the project has grown, the code base has become difficult to maintain and read. As the project's sole developer, I am struggling to find solutions to make the code more readable, clear, and maintainable.

If you have experience with lex and yacc parser or you are not afraid of challenges and have some free time, I would greatly appreciate your help and ideas. I have attempted refactoring multiple times, but each time I have found that my solution has resulted in a similar mess to the existing code base.

I understand that this is a volunteer effort, but as a thank you, I am happy to offer a pizza and include your name in the project with a special thank you in the README file. The main goal of refactoring is to make the code base easy for other developers to understand, so that they can add new features on their own.

Thank you for your support and I look forward to working together to improve the project

[xnuinside]

The main issue with the parser structure is that it first splits the text input into tokens based on regular expressions (as shown in this example: https://github.com/xnuinside/simple-ddl-parser/blob/main/simple_ddl_parser/ddl_parser.py#L161 and some tokens are defined as words for 1-to-1 matching https://github.com/xnuinside/simple-ddl-parser/blob/main/simple_ddl_parser/tokens.py). After that, it tries to find the necessary sequence of tokens based on defined rules, and if it finds this sequence, it enters the method.

The sequence of tokens is defined in comments within functions, like this example: https://github.com/xnuinside/simple-ddl-parser/blob/main/simple_ddl_parser/dialects/sql.py#L13

In this example, expr represents the previous expression result that was parsed. Upper case letters are tokens and lower case letters are links to other sequences.

For example, the link means that pid is also a sequence, defined in another method, here: https://github.com/xnuinside/simple-ddl-parser/blob/main/simple_ddl_parser/dialects/sql.py#L1348

When the parser meets this order of tokens and sequences, it enters the function and executes the code. The result of parsing this sequence is always put in p[0] (this is how the parser ply works).

And as you can see from the code base, there are a lot of functions that implement the parser logic and produce the output. However, the main problem is that I am struggling to find a way to better structure them, group them, or refactor them to make them more clear and easy to understand. Given the parser's limitations I described earlier, it isn't easy to find a solution.
Of course, many parts of the code can be moved to some utility functions, but they are not reusable. However, I am breaking down large functions into smaller chunks to make them more manageable.

[xnuinside]

Also right now all grammars for different dialects are messed up and possible to start from separating each dialect as parser class, so it will be like several different parsers for each dialect, inherited from main SQL. And if user knows dialect - he provided it as argument, if not - under the hood will be method that iterate over the dialect parser and find the one, that works better, but it can be much more slow than now.

[mrmonkington]

Hi @xnuinside, I agree the parser needs refactoring to disentangle all of the dialects. In trying to solve #180 I have found it very difficult due to conflicting test cases (e.g. key needs to be a reserved word for mysql but is used as an identifier without escaping)

Would this be easier if Ply was replaced with Sly (successor by same author)? It seems to more naturally support subclassed parsers, and also removes the need for parsetab.py cache, which might otherwise complicate this refactor?

Thank you for all of your work on this project - I will try to help as much as I can!!

[xnuinside]

@mrmonkington , I was out for a long time because of personal issues. I will check Sly - thank you for mentioned it, I will take a look