Make parser nesting easier

[rumkin]

Allow parsers to nest inside each other. Let assume I've created expression parser and want to built it into different template engines using special sequences:

Template string with placeholder: {{= 1 + 1 }}
Template string with placeholder: <?= 1 + 1 ?>

The problem string {{= "a" + "}}" }} : template parser should to know that "}}" is a string and it's not a substitution end.
The same problem string: <?= "a" + "?>" ?>

To do this now I should rewrite and recompile template using peg rules of my expressions parser. Why I could not to use complied parser.

I think that actually }} and ?> should became the parse ending rule (like EOF) for expressions parser. This allow us preserve parsing location() for embedded languages, reuse existing parsers code efficiently and make languages cross embeddable.

[dmajda]

@rumkin Do I understand correctly that you’d like to seamlessly use an existing, compiled parser from another parser to parse a nested expression terminated by a specific string (or, more generally, an expression)?

To me, it makes more sense to include and reuse just the inner parser grammar rules (imported into the outer grammar using some mechanism, see #38), not the compiled parser. Would that work for your use case? If not, why? (Note the need of passing and respecting the terminating string/expression is common to both approaches.)

[rumkin]

It look like #38 but without compilation. The benefit of this behaviour is to allow combine parsers without recompilation using ready npm modules.

[dmajda]

@rumkin I see.

Is this an actual problem you bumped into (e.g. you had an actual parser you wanted to reuse in some real-world project, but couldn’t) or is this something you just think would be nice to have? (I’m trying to assess how relevant and important this feature is.)

Also, how would your ideal solution look like? Specifically, how would you imagine the portion of the outer parser that uses the inner parser to look like?

[rumkin]

Yes, I had expression parser which is using in 3 separated projects. And it easer to use npm to manage parser as dependency.

And I think it easer to implement than #38. Parser should make the end rule configurable and add subparser method to stack parsers and to preserve location.

Code example:

import 'someparser'

substitution = subStart body:#someparser(subEnd) subEnd

subStart = "{{"

subEnd = "}}"

Other variant is:

substitution = subStart body:#expressionParser(subEnd) subEnd

subStart = "{{"

subEnd = "}}"

const parser = require('./parser.js');
parser.expressionParser = require('someparser');

[dmajda]

Thanks.

Importing compiled parsers instead of grammars is certainly simpler to manage via npm (the author of imported parser can just distribute it as usual, without including its source grammar). On the other hand, it’s much less powerful mechanism. For example, once parametrized rules are implemented (#45), it would be useful for grammars to be able to export rule templates which the importing grammar can use and fill-in. This wouldn’t be easily possible with compiled parsers.

Allowing importing both grammars and compiled parsers would be a solution, but I’d like any import mechanism to be simple, so this isn’t a direction I’d prefer. If I were to choose right now, I’d just go for plain grammar imports — in other words, I’d trade more power for somewhat less convenient grammar management.

I’m marking this issue as post-1.0.0 and will keep it open for now. Additional insights, use cases, and votes are welcome.