Formatter cannot handle tabs and spaces properly for hidden terminal rules

[joluj]

Langium version: reproducible in 1.2.0 & 2.1.0 (current)
Package name: langium

Steps To Reproduce

1. Open example/domainmodel in vscode with the domainmodel extension enabled
2. Add a new line with tabs and spaces before a comment (i.e. a hidden terminal rule)
3. Format code

Code example:

qualified-names.dmodel

This file contains tabs and spaces mixed. It works fine with non-hidden rules but breaks with hidden rules (e.g. comments)

entity E1 {
	name: String // Tabs are replaced in non-hidden nodes (expected)
  	  time: big.Int // Tabs and spaces mixed are converted to spaces (expected)
	// Tabs are not replaced in hidden nodes
 	  // Tabs and spaces mixed lead to total confusion
}

The current behavior

• Tabs are not converted to spaces in hidden rules.
• Indent breaks if tabs and spaces are mixed in hidden rules.
• Both bugs don't occur with non-hidden rules.

entity E1 {
-	name: String // Tabs are replaced in non-hidden nodes (expected)
-  	  time: big.Int // Tabs and spaces mixed are converted to spaces (expected)
-	// Tabs are not replaced in hidden nodes
- 	  // Tabs and spaces mixed lead to total confusion
+    name: String // Tabs are replaced in non-hidden nodes (expected)
+    time: big.Int // Tabs and spaces mixed are converted to spaces (expected)
+	// Tabs are not replaced in hidden nodes
+ // Tabs and spaces mixed lead to total confusion
}

The expected behavior

• Hidden and non-hidden rules have the same behavior

entity E1 {
-	name: String // Tabs are replaced in non-hidden nodes (expected)
-  	  time: big.Int // Tabs and spaces mixed are converted to spaces (expected)
-	// Tabs are not replaced in hidden nodes
- 	  // Tabs and spaces mixed lead to total confusion
+    name: String // Tabs are replaced in non-hidden nodes (expected)
+    time: big.Int // Tabs and spaces mixed are converted to spaces (expected)
+    // Tabs are not replaced in hidden nodes
+    // Tabs and spaces mixed lead to total confusion
}

[georg-schwarz]

Is there already any known workaround or solution for this?

[msujew]

@georg-schwarz Someone would probably need to take a look at how edits for hidden nodes are created. I don't think there's a fix for this yet. I have a general refactoring of the formatter in mind, which is why this issue is currently in the backlog.

[georg-schwarz]

Thanks for the pointer!
We will share our solution here in case we decide to tackle it before you are done with the refactoring you mentioned ;-)