processing

Data Processing Toolkit in GluonNLP

We provide a bunch of data

Clean and Tokenize a Parallel Corpus

To clean and tokenize a parallel corpus, use

nlp_process clean_tok_para_corpus --help

Learn a subword model

To learn a subword tokenizer, use

nlp_process learn_subword --help

Apply the learned subword model

To apply the learned subword tokenizer, user

nlp_process apply_subword --help