Capturing nested comments

[DiKetarogg]

Goals

• Capture comments including all whitespace characters.
• Allow nesting of comments.

Code so far

struct comment {
	struct multiline {
		static constexpr
		auto rule = capture(token(
			dsl::brackets(LEXY_LIT("/*"), LEXY_LIT("*/"))
			.list((peek(LEXY_LIT("/*"))
	  			>> dsl::recurse_branch<multiline>)
	  			| dsl::code_point)))
		;
		static constexpr
		auto value = ::lexy::as_string<std::string>;
	};
	static constexpr
	auto rule =
		capture(token(
			LEXY_LIT("//") >> identifier(dsl::code_point - dsl::ascii::newline)
		))
		| dsl::p<multiline>
	;
	static constexpr
	auto value = ::lexy::as_string<std::string>;
};

Problems:

The code above produces the following error at runtime:

output.s: /app/example.cpp:20117:
static bool lexyd::_recb<comment::multiline>
::_depth_handler<lexy::sink_parser>
::increment_depth(Context &, Reader &)
 [Production = comment::multiline
  , DepthError = void
  , NextParser = lexy::sink_parser
  , Context = lexy::_pc<
   lexy::_mh
   , void
   , lexyd::_token<
    lexyd::_br<
     lexyd::_lit<char, '/', '*'>
     , lexyd::_lstt<
      lexyd::_lit<char, '*', '/'>
      , lexyd::_chc<
       lexyd::_br<lexyd::_peek<lexyd::_lit<char, '/', '*'>, void>
        , lexyd::_recb<comment::multiline>
       >
       , lexyd::_cp<void>
      >
      , void, lexyd::_reco<void, lexyd::_lit<char, '*', '/'>>
     >
    >
   >::_production
  >, Reader = lexy::_br8
 ]: Assertion
 `(control_block->max_depth > 0)
  && ("dsl::recurse_branch<P> is disabled in this context")' failed.
Program terminated with signal: SIGSEGV

Attempted:

• Adding max_recursion_depth variable
• Using recurse instead of recurse_branch

[foonathan]

Your issue is that you use dsl::token inside the dsl::capture. You can then no longer use recursion at all for technical reasons.

You can avoid that by using the second overload of dsl::capture which takes a token production: https://lexy.foonathan.net/playground/?id=5oeabd1ob&mode=tree

Note that multiline and single do not produce values on their own, only the final comment will produce a span that contains the entire comment. I'm assuming this is the semantic you wanted? I'm not sure how you want to capture nested comments otherwise.

[DiKetarogg]

Note that multiline and single do not produce values on their own, only the final comment will produce a span that contains the entire comment.

I get errors about missing member value for comment and multiline and single. After adding value, I get a static assert error: missing value callback overload for production.

I'm assuming this is the semantic you wanted?

Yes.

I'm not sure how you want to capture nested comments otherwise.

Me neither.

[foonathan]

I get errors about missing member value for comment and multiline and single

right. Since you don't want any values during parse, you need to add static constexpr auto value = lexy::noop.

[DiKetarogg]

Since you don't want any values during parse

Actually, I want comments to be captured as a string. I'm translating my language to another existing language and for that reason I want the end result to keep the comments.

[foonathan]

Yes, so you need to add static constexpr auto value = lexy::noop to multiline and single, but not comment - there you need a callback that handles the captured lexeme.