Multi-Criteria Chinese Word Segmentation with Transformer Encoder

A Pytorch(fastNLP) implementation for "A Concise Model for Multi-Criteria Chinese Word Segmentation with Transformer Encoder"

Code explaination

First, place the raw data at data/ and prepare corpora using :

python prepoccess.py

Then prepare the inputs for training CWS model:

python makedict.py
python make_dataset.py --training-data data/joint-sighan-simp/bmes/train-all.txt --test-data data/joint-sighan-simp/bmes/test.txt -o <output_path>

It will generate a .pkl file as <output_path>. It contains a dict in the following format:

{
    'train_set': fastNLP.DataSet
    'test_set': fastNLP.DataSet
    'uni_vocab': fastNLP.Vocabulary, vocabulary of unigram
    'bi_vocab': fastNLP.Vocabulary, vocabulary of bigram
    'tag_vocab': fastNLP.Vocabulary, vocabulary of BIES
    'task_vocab': fastNLP.Vocabulary, vocabulary of criteria
}

Finally, train the model using (freezing the embeddings):

python main.py --dataset <output_path> --task-name <save_path_name> \
--word-embeddings <file_of_unigram_embeddings> --bigram-embeddings <file_of_bigram_embeddings> --freeze --crf --devi 0

The embedding files can be found here.

(*merge.txt denotes both simplified and traditional Chinese while *corpus.txt contains simplified Chinese only)

Continue to train the model without freezing the embeddings:

python main.py --dataset <output_path> --task-name <save_path_name> --num-epochs 20 --old-model result/<save_path_name>/model.bin \
--word-embeddings <file_of_unigram_embeddings> --bigram-embeddings <file_of_bigram_embeddings> --step <previous_training_step> --crf --devi 0

More details about commands can be found by using:

python main.py --help