TokenType structure and API

[bd82]

The TokenType structure has many optional properties.

export interface TokenType {
    name: string
    GROUP?: string
    PATTERN?: TokenPattern
    LABEL?: string
    LONGER_ALT?: TokenType
    POP_MODE?: boolean
    PUSH_MODE?: string
    LINE_BREAKS?: boolean
    CATEGORIES?: TokenType[]
    tokenTypeIdx?: number
    categoryMatches?: number[]
    categoryMatchesMap?: {
        [tokType: number]: boolean
    }
    isParent?: boolean
    START_CHARS_HINT?: (string | number)[]
}

I believe this is due to legacy reasons as in the past TokenTypes could be classes with static properties. The fact this structure only has a single mandatory property (name) means that any API that accepts TokenType (e.g CONSUME) could also accept (by mistake) other similar structures. For example a JS Function has name property and is therefore acceptable anywhere a TokenType is accepted.

I believe the TokenType structure should more closely match what is returned by "createToken"
which means a-lot less if any optional properties.

Also getting rid of the upper case property names would also be advisable.

[bd82]

By definition this is a breaking change, however if the creation of TokenTypes is always done by
calling createToken, does it matter if the API gets broken? does it really mandate an new major version?

[bd82]

I think this structure may actually be missing one of the more important properties (tokenIdx/Idx)

[bd82]

It may be a good idea to "hide" some of the internal properties using symbols instead of string properties.