Whitespace not properly ignored in digit sequences

[zhmu]

Hi all!

TL;DR Why is whitespace still part of the result even though I have a silent WHITESPACE rule, and how would I get rid of it?

I'm trying to parse the following input (Rust 1.73.0, Pest 2.7.7):

locals { 1 = 2, 3 = 4 }

My test program (source code is below) yields the following, note that it includes whitespace:

>> '1   ' = '2'
>> '3 ' = '4 '

My Pest file contains:

WHITESPACE = _{ " " | NEWLINE }

NUMBER = { ASCII_DIGIT+ }
local_def = { NUMBER ~ "=" ~ NUMBER }
locals = { "locals" ~ "{" ~ local_def ~ ("," | local_def)* ~ ","? ~ "}" }

My Rust code is the following:

use pest::Parser;
use pest_derive::Parser;

#[derive(Parser)]
#[grammar = "example.pest"]
pub struct TestParser;

fn main() {
    let s = "locals { 1   = 2, 3 =   4 }";
    let p = TestParser::parse(Rule::locals, &s);
    let p = p.unwrap().next().unwrap();
    for local in p.into_inner() {
       let mut local = local.into_inner();
            let num = local.next().unwrap();
            let value = local.next().unwrap();
            println!(">> '{}' = '{}'", num.as_str(), value.as_str());
    }
}

I expected the whitespace to be stripped from the result, but it turns out it isn't. My goal is to place the num/value in a data structure.

I'm sure I am doing something wrong. Can someone help me?

[zhmu]

Mmm, this seems to be the same problem as stated in #396. Changing the Pest file to:

NUMBER = @{ ASCII_DIGIT ~ (ASCII_DIGIT)* }

Seems to prevent the issue.

[zhmu]

It turns out that you need to opt-in the grammar-extras feature for pest_derive:

pest_derive = { version = "2.7", features = ["grammar-extras"] }

I understand the need to keep old grammars working, but perhaps this should be added to the documentation in some way: I'd expect you to want this behaviour for new projects.

Finally, apologies for raising a new discussion and not finding the prior discussions/issues. I guess I didn't look hard enough.

[NoahTheDuke]

No worries, thanks for posting the solution. You're right that we should update the docs to point this out. We have stability guarantees we have to keep, otherwise this is an obvious beneficial change.