Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parsing ends prematurely #1868

Open
yanny7 opened this issue Oct 20, 2022 · 1 comment
Open

Parsing ends prematurely #1868

yanny7 opened this issue Oct 20, 2022 · 1 comment

Comments

@yanny7
Copy link

yanny7 commented Oct 20, 2022

I encountered problem with recovery, when in specific case recovery ends on error and can't continue (it fails in OR alternative).

working example:

(function jsonGrammarOnlyExample() {
  // ----------------- Lexer -----------------
  const createToken = chevrotain.createToken;
  const Lexer = chevrotain.Lexer;

  const IF = createToken({name: "IF", pattern: /\bIF\b/});
  const THEN = createToken({name: "THEN", pattern: /\bTHEN\b/});
  const GOTO = createToken({name: "GOTO", pattern: /\bGOTO\b/});
  const Assign = createToken({name: ":=", pattern: /:=/});
  const Colon = createToken({name: "Colon", pattern: /:/});
  const EOL = createToken({name: "EOL", pattern: /\n|\n\r|\r\n|\r/y, line_breaks: true});
  const ObjName = createToken({name: "ObjName", pattern: /[a-zA-Z_]+/});
  const WhiteSpace = createToken({
    name: "WhiteSpace",
    pattern: /[ \t]+/,
    group: Lexer.SKIPPED
  });

  const jsonTokens = [WhiteSpace, IF, THEN, GOTO, EOL, Assign, Colon, ObjName];

  const JsonLexer = new Lexer(jsonTokens, {
    // Less position info tracked, reduces verbosity of the playground output.
    positionTracking: "onlyStart"
  });


  // ----------------- parser -----------------
  const CstParser = chevrotain.CstParser;

  class JsonParser extends CstParser {
    constructor() {
      super(jsonTokens, {
        recoveryEnabled: true
      })

      const $ = this;

      $.RULE("esl", () => {
        $.MANY(() => $.SUBRULE($.actions));
      });
        
      $.RULE("actions", () => {
        $.OR([
          {ALT: () => $.SUBRULE($.if)},
          {ALT: () => $.SUBRULE($.label)},
          {ALT: () => $.SUBRULE($.assignment)},
          {ALT: () => $.CONSUME(EOL)}
        ]);
      });
      
      $.RULE("if", () => {
        $.CONSUME(IF);
        $.OR1([
          {ALT: () => $.CONSUME(THEN)},
          {ALT: () => {
            $.CONSUME(GOTO);
            $.CONSUME(ObjName);
          }}
        ]);
        $.CONSUME(EOL);
      });
      
      $.RULE("label", () => {
        $.CONSUME(ObjName);
        $.CONSUME(Colon);
        $.CONSUME(EOL);
      });
      
      $.RULE("assignment", () => {
        $.CONSUME(ObjName);
        $.CONSUME(Assign);
        $.CONSUME(EOL);
      });

      // very important to call this after all the rules have been setup.
      // otherwise the parser may not work correctly as it will lack information
      // derived from the self analysis.
      this.performSelfAnalysis();
    }

  }

  // for the playground to work the returned object must contain these fields
  return {
    lexer: JsonLexer,
    parser: JsonParser,
    defaultRule: "esl"
  };
}())

when you type (don't forget newline after last sentence)

IF THEN
IF THENs
IF THEN

recovery stops on second line.

Problem is that RecognizerEngine fails to find path (in orInternal function) and thus its considered as stuck in manyInternalLogic.

Maybe it should skip next token and try to recover again. Or in my case, I would like to tell recovery that skip up to newline and try to continue with parsing from there.

@yanny7 yanny7 changed the title Recovery ends prematurely Parsing ends prematurely Oct 20, 2022
@yanny7
Copy link
Author

yanny7 commented Oct 20, 2022

Note: after removing alternation label or assignment from actions rule recovery succeed and parser continues correctly

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant