Optionally preserve unrecognized tokens

[myedibleenso]

The tokenizer cannot handle just any character sequence. There are scenarios where a user would like to preserve these symbols in the Document that is produced.

I haven't had a hand in the implementation of the tokenizer, but is it possible to reinsert these symbols as tokens in the doc we return? lemmata, tags, etc. should probably default to some sort of UNK tag.

Related: if processors moves to a transformers backbone for annotation as planned, will the tokenizer be replaced by a wordpiece tokenizer or will predictions be re-mapped to the word-like tokens recognized by the current tokenizer?

[MihaiSurdeanu]

• We will preserve our word-level tokenizer, and then use subword tokenization just within words. So, from the outside, the API stays the same.
• I have mixed feelings about preserving weird symbols. Tokenization and simpler tasks may work, but this will mess up parsing for sure...

[myedibleenso]

I have mixed feelings about preserving weird symbols. Tokenization and simpler tasks may work, but this will mess up parsing for sure...

Agreed. It needs to be done in a way that doesn't affect annotation (see #681)

[myedibleenso]

Although if processors moves to using transformers, re-introducing tokens after annotation might be confusing when inspecting things like attention weights.

[kwalcock]

See #716 and #290. The UNK value is presently an empty string.