Question: parsing line-based tokens (multiple character lookahead)

[viktomas]

Hello 👋

My question: Is it possible to parse the following grammar without implementing a custom lexer?

I'm trying to write a parser for a grammar similar to the tree-sitter test file format.

File:

===
pages/file.md
---
content
===
pages/another.md
---
- another content
- with multiple
- lines

The grammar would be:

testFiles = testFile*
testFile = "===" filePath "---" content

where filePath is a line of text and content is any string except for === (which signifies the start of the new file.

I got this far:

type TestFile struct {
	Path    string `FileStart@Content`
	Content string `ContentStart@Content`
}

type Example struct {
	Input []TestFile `@@*`
}

var (
	exampleLexer = lexer.MustSimple([]lexer.SimpleRule{
		{`FileStart`, `===`},
		{`ContentStart`, `---`},
		{"whitespace", `\s+`},
		{"Content", `[^\n]+`},
	})
	exampleParser = participle.MustBuild[Example](
		participle.Lexer(exampleLexer),
	)
)

But because I defined content as [^\n]+, the parser can't parse multi-line file content. And since Go regexp doesn't support lookahead, it's probably impossible to use SimpleRules to parse this grammar, right?

I was excited about defining the grammar in struct field tags, but for this particular case, it might be simpler/easier to write custom code that reads the input line-by-line. WDYT?

[alecthomas]

I'd probably do something like:

type TestFile struct {
	Path    string `FileStart @(Content '\n'?)+`
	Content string `ContentStart @(Content \n'?)+`
}

[viktomas]

Thanks @alecthomas you helped me heaps! That was almost the solution 🙇. Since I care about the new lines in the Content field, I had to make the new line an explicit symbol like this

type TestFile struct {
	Path    string `FileStart NewLine @Content NewLine`
	Content string `ContentStart NewLine @(Content NewLine?)+`
}

var (
	exampleLexer = lexer.MustSimple([]lexer.SimpleRule{
		{`FileStart`, `===`},
		{`ContentStart`, `---`},
		{"NewLine", `[\n]`},
		{"Content", `[^\n]+`},
	})
	exampleParser = participle.MustBuild[Example](
		participle.Lexer(exampleLexer),
	)
)

Explanation for other people who might look at this:

Since the lexer tries the rules in order, each line will try the FileStart and ContentStart rules before the more generic Content rule.

I thought I had to parse the whole content as one token, but thanks to @(Content NewLine?)+ it's possible to scan many "content lines" and let the parser group the Content and NewLine tokens together into one field.