You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I think StreamTagger can be removed since it is not refereed from any other codes. At the same time, several legacy codes can refactor to modernize the code. For example, Gosen should utilize char.def which is an external configuration file that can specify how to handle characters by its type, which is hard-corded in Gosen at this moment, and that enable us to freely modify the tokenization behavior.
The text was updated successfully, but these errors were encountered:
I think StreamTagger can be removed since it is not refereed from any other codes. At the same time, several legacy codes can refactor to modernize the code. For example, Gosen should utilize char.def which is an external configuration file that can specify how to handle characters by its type, which is hard-corded in Gosen at this moment, and that enable us to freely modify the tokenization behavior.
The text was updated successfully, but these errors were encountered: