Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Matching token values??? #886

Open
InfiniteCoder01 opened this issue Apr 14, 2024 · 2 comments
Open

Matching token values??? #886

InfiniteCoder01 opened this issue Apr 14, 2024 · 2 comments

Comments

@InfiniteCoder01
Copy link

Let's say I use logos for lexer:

extern {
    type Location = usize;
    type Error = Error;

    enum Token {
        "ident" => Token::Ident(<String>),
    }
}

And now I want to match for a return expression:

pub Expression: ir::Expression = {
    Block => ir::Expression::Block(<>),
    "ident" <Expression> => ir::Expression::Return(Box::new(<>)),
}

There is no way to do so. And I have to use ident token, since I want to be able to declare a variable named "return".

@yannham
Copy link
Contributor

yannham commented Apr 15, 2024

If return is a keyword of your language, it's customary to lex it as a separate token rather than consider it as an identifier, in particular because it may be ambiguous in some cases (that's why, in most language, you can't use keywords as identifier).

Otherwise, if it can never be ambiguous and you want to keep return as a valid identifier, one possibility is to match on the ident from within the action code. That is, if you want to mirror something like (beware, the following code is NOT possible in current LALRPOP, that's an ideal):

<id: "ident"> <exp: Expression> when id == "return" => ir::Expression::Return(Box::new(exp))

You can approximate it with:

<id: "ident"> <exp: Expression> => {
  if id == "return" {
    ir::Expression::Return(Box::new(exp))
  }
  else {
    panic!("expected return, got {id}")
  }

Of course, in a real use case, you might not want to panic but to rather return an error. This is possible in LALRPOP via fallible actions and custom parse errors. Or even simpler if you have an error node in your AST.

@yannham
Copy link
Contributor

yannham commented Apr 15, 2024

Yet another possibility is to make return a special token of your lexer, so that you can write your rule like:

"return" <Expression> => ir::Expression::Return(<>)

and to have an overarching identifier rule that includes "return":

Identifier: String = {
  "ident" => <>,
  "return" => "return".to_owned(),
}

Then use Identifier everywhere in place of "ident".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants