Whitespace and comment handling in lexer

[mikeclayton]

Whitespace and comments are consumed and emitted by the lexer, but not mandatory between some tokens. This means that, for example, the following text:

12345myIdentifier

gets converted to two valid tokens:

IntegerLiteralToken: 12345
IdentifierToken:     "myIdentifier"

This probably isn't much of a problem as in general sequences of tokens without whitespace between them aren't valid in any parser rules anywhere this could appear anyway, but it would probably be better if the input text failed at the lexer stage rather than the parser when mandatory whitespace is missing.