Whats a good way to only parse certain parts of text?

[RobertCraigie]

For example, I want to parse input that looks like this

Arbitrary stuff
@Foo(...)

or

Arbitrary stuff @Foo(...)

But I only care about:

1. If there is an @Foo in the text
2. If the contents encapsulated within the () following the @Foo() are valid

Is there a builtin way to do this easily with Lark? If not, is there a better approach than to just regex match for @Foo(...) and only pass that to Lark?

[MegaIng]

See this small example I created: #1371 (comment) It's not currently built into lark, but I created a small proof-of-concept that adds it to lark. If you find it useful, we could look into adding it.

[RobertCraigie]

Just tried it and it looks like it's almost exactly what I was looking for. The only other case is how to handle syntax errors as I would want an error to be raised if the contents inside the @Foo(...) are invalid but I can appreciate how that's difficult... Is there a specific way I should structure my grammar for that to just work™️?

Or maybe another approach here, at least for my case, is to have two grammars, one to just scan for @Foo(...) and the other to include parsing the contents?

[MegaIng]

The problem is that there somehow needs to be a decision at which point this construct looks enough like something we are supposed to parse, called a "pin". Those kinds of anchors are generally better integrated into PEG parsers and I am not really sure how that could be added to lark, maybe @erezsh has an idea.

[erezsh]

Tell me if I'm being too naive, but why not set a "pin" on the number of matched tokens. So, above 'n' tokens it becomes a legit parse and errors are raised. The user can give it as a parameter to scan(), and they can adjust it based on a sample text, until it gets enough results without error.

Or, we can specify a specific token (by name) that will serve as the pin.

For an actual pin implementation, it might be possible to set a mark as an empty rule, and the scanner could collect a list of all the parser states following a reduce on that empty state, and then compare to those states to see if the pin has been reached.

But I would start with the much simpler ones first :)

[MegaIng]

Tell me if I'm being too naive, but why not set a "pin" on the number of matched tokens. So, above 'n' tokens it becomes a legit parse and errors are raised. The user can give it as a parameter to scan(), and they can adjust it based on a sample text, until it gets enough results without error.
Or, we can specify a specific token (by name) that will serve as the pin.

These are basically the same ideas I have come up with as well, and I guess they might work well enough, but they definitely feel half-finished.

For an actual pin implementation, it might be possible to set a mark as an empty rule, and the scanner could collect a list of all the parser states following a reduce on that empty state, and then compare to those states to see if the pin has been reached.

An empty marker rule is an interesting idea, I might play around with that. This might also be useful for better error messages if integrated into the match error system we have.

But I would start with the much simpler ones first :)

I think I will first polish up a version with no pining at all and then try around a bit. But either way, I can't do it now because I have an exam in two days.

[erezsh]

Good luck!