feat: add custom cell magic parser

[plamut]

Fixes #166.
Fixes #108.

This PR is to preview the custom cell magic line parser.

PR checklist

• Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
• Ensure the tests and linter pass
• Code coverage does not decrease (if any source code was changed)
• Appropriate docs were updated (if necessary)

[plamut]

@shollyman I would like to hear your thoughts on this.

The current cell magic line parser is not aware of Python structures (dicts, lists ...) and can thus incorrectly parse the --params option (example issues: #166 and #108). The default parser breaks these structures on whitespace, and while we stitch these parts back together and operate on the result, this only happens after the parsing has already taken place and bugs have sneaked in.

I took a stab at it and came up with a POC custom parser that successfully parses non-trivial input lines such as the following:

%%bigquery    target_var --use_legacy_sql --project my.project.name --params {'nested_dict': {'bar': 'baz', 'some_tuple': (-1, 3.2,)}, "answer": -42, 'array': ['aaa bbb', ["nested", 'list'], 'ccc ddd'], '--params': 'tricky --inner-option string literal',} --verbose  

The following test snippets demonstrates that the parser correctly recognized the options:

from google.cloud.bigquery.ipython_magics import line_arg_parser as lap

example_line = (
    "  target_var "
    "--use_legacy_sql "
    "--project my.project.name "
    "--params {'nested_dict': {'bar': 'baz', 'some_tuple': (-1, 3.2,)}, "
    "\"answer\": -42, 'array': ['aaa bbb', [\"nested\", 'list'], 'ccc ddd'], "
    "'--params': 'tricky --inner-option string literal',} "
    "--verbose  "
)

lexer = lap.lexer.Lexer(line)
parser = lap.parser.Parser(lexer)
parse_tree = parser.input_line()

printer = lap.visitors.TreePrinter()
printer.visit(parse_tree)

Output:

Input:
    destination_var:    target_var
    Command options:
        --use_legacy_sql 
        --project my.project.name
        --params {'nested_dict': {'bar': 'baz', 'some_tuple': (-1, 3.2)}, "answer": -42, 'array': ['aaa bbb', ["nested", 'list'], 'ccc ddd'], '--params': 'tricky --inner-option string literal'}
        --verbose 

The custom parser could be used to extract out the --params <...> part, evaluate it separately, and only feed the remainder of the input to the default parser.

Do you think we should pursue this further?

The grammar used is of course a simplification that is aimed to be "good enough" to reliably extract the --params part in typical uses cases, but it does not try to be bullet-proof. For instance, it currently does not support non-string dict keys, sets, floats in exponential notation, etc., although that can all be added depending on how much we want to complicate. :)

---

cc: @tswast Do customers complain about the default parser's limitations often? Do you think there even exist reasonable use cases where --params`dictionary would be incredibly complex?

[tswast]

Do customers complain about the default parser's limitations often?

I wouldn't say often, but it has come up more than once

[cguardia]

This looks pretty good. Thanks.

[cguardia]

I think this is looking good.

[tswast]

This is awesome! It does require a bit of refactoring, though. We need to keep google.cloud.bigquery.magics module/package name due to google.cloud.bigquery.magics.context

I'd also like to see if we can lean on pyparsing for a lot of the lexing / parsing work. I worry that those modules are going to be particularly challenging to maintain without either "literate programming" level comments or relying on common modules to make the parser easier to understand.

[tswast]

Thanks so much for this explanation and the clearer state names!

Fun fact: anything that can be parsed with a regular expression can also be parsed by a finite state machine and vice versa. It's called a regular language. That means we might have been able to convert this into one giant regular expression, but I like having these explicit states better.

I recall that lookahead & backtracking isn't a regular expression in the computer science definition, so I don't think this is technically a "regular" language, though.

[plamut]

Indeed, regular expressions correspond to finite automata, although the regex implementation is a bit more powerful than that (AFAIK), as it supports things like lookahead and lookbehind.

The language itself recognized by the parser is not regular, true, as the grammar allows for recursive patterns such as nested dicts. A proof of non-regularity is left as an exercise to the reader. 😆

(hint: pumping lemma).

[tswast]

Great work! Thanks for the clear explanations.