Expose information about indentation

[tgodzik]

Currently tree.tokenize retokenizes the code within the pos of the tree, which means that those tokens are not related to the ones used in the parser and we will not get any information about indentation. That information might particularly be useful to tools that need to deal with indentation for example scalafmt. There are ways to work around it, but they cause significant complexity.

It would be cool to somehow expose information about where indentation was detected. Some options:

1. Keep additional isIndented method in the Tree
2. Safe the tokens from parsing while also inserting appropriate Indentation.Indent and Indentation.Outdent tokens
3. Just safe scanner tokens inside ScalametaParser, so that some tools can use that explitely.

1

This is something that we haven't done for most cases and introduced additional Trees for anything that can have an additional Boolean value. However, we could argue that isIndented is similar to for example pos, in the sense that it can be needed in every Tree. This would not solve an issue with the tokens, but would mean that we wouldn't need to save the tokens from parsing.

2

This requires two things:

• keeping the tokens, which is something that was not done previously and most likely for performance/memory reasons
• creating a new end tokens collections accessible via the tokens.method we could just save each token after next() method

3

ScalametaParser does not need to be binary compatible and is not exposed to the users, so this would mean that we would introduce a level of coupling between Scalameta and any tools that would use it explicitely.

None of the above seems like the best solution, I would turn to 2., but I am unsure of the consequences to any kind of optimizations.