How to access hidden CST tokens from AST?

[enikao]

I'm trying to retrieve the complete input text for a specific AST node, including all hidden tokens.

My example is a properties file:

integerProp = 1
myString = "Hello, world!"

So if I accessed AST model.props[0].$cstNode.content, I'd expect to see 5 tokens: ID, WS, EQ, WS, INT. But the two WS tokens are omitted.
I also tried to poke around in the model.$document structure, to no avail.
They seem to exist in lexerResult.hidden, but not propagated further?

[msujew]

Hey @enikao,

unlike in Xtext, whitespace tokens aren't actually stored in the CST and simply discarded. All other hidden tokens can be accessed via the CST though. You can change that behavior by overriding the TokenBuilder and change the group of the white space token to 'hidden'.

[cdietrich]

if you do a document.getText(preNonHidden.line/char, nextNonHidden.line/char) it should (not exact api)

[enikao]

ah ok, I get it. So it would be a mixture of offset calculation and text slicing.

[szarnekow]

that information can easily be used to infer whitespace information without requiring to store whitespace explicitly

It's a deliberate decision to ignore windows line endings/unix line endings during full serialization?

[cdietrich]

there is no serialization, is there?

[msujew]

there is no serialization, is there?

Right, there is no serialization yet.