Mask certain tokens during annotation

[myedibleenso]

There are many cases where a token is better off invisible to a sequence tagger or shift-reduce parser (ex. the bullet in a bulleted list, ®, °, etc.). If such symbols have not been seen in training, they may have unexpected effects on the output sequence. It would be convenient to provide masked tokens for specific steps in the annotation pipeline using a regular expression or list of strings.

[kwalcock]

This is with the veil:

This is without:

Here's the extra code:

    val veil = text.indices
        .filter { index =>
          val char = text.charAt(index)
          char == '®' || char == '*'
        }
        .map { index => index to index }
    val veiledText = new VeiledText(text, veil)
    val document = processor.annotate(veiledText.mkDocument(processor))

[MihaiSurdeanu]

Nice!