transformer example and texar-pytorch/bin/utils should be improved

[gpengzhi]

In texar-pytorch/bin/utils, sentencepiece is a package instead of a tokenization method, so sentencepiece encoding is not a very accurate way to describe the tokenization method. sentencepiece includes two sub-word tokenization methods: byte-pair-encoding (BPE)[Sennrich et al.] and unigram language model [Kudo.].

There is no Word Piece Model (WPM) pipeline in sentencepiece, and the code we used

spm_train --input=train.src,train.tgt --vocab_size 32000 --model_prefix=wpm-codes

is actually using unigram language model. Here, Unigram is the default method.

transformer example need to be updated accordingly.