Help: How to Load pre-trained chinese word embedding ?

[xiaoxiaoAurora]

How to load a local and pre-trained Chinese word vector file?

[szha]

For Chinese, these fasttext embeddings are available:
wiki.zh, wiki.zh_classical, wiki.zh_min_nan, wiki.zh_yue, cc.zh.300
You can load them with gluonnlp.embedding.FastText('wiki.zh'). char-ngram embeddings are also available that can be automatically downloaded with gluonnlp.embedding.FastText('wiki.zh', load_ngrams=True)

Note that the embedding files are very large, so the one-time download could take some time. If you are in China and run into connection issue, try export MXNET_GLUON_REPO=https://apache-mxnet.s3.cn-north-1.amazonaws.com.cn/ before starting the process to use the Gluon repo mirror in China

[xiaoxiaoAurora]

For Chinese, these fasttext embeddings are available:
wiki.zh, wiki.zh_classical, wiki.zh_min_nan, wiki.zh_yue, cc.zh.300
You can load them with gluonnlp.embedding.FastText('wiki.zh'). char-ngram embeddings are also available that can be automatically downloaded with gluonnlp.embedding.FastText('wiki.zh', load_ngrams=True)

Note that the embedding files are very large, so the one-time download could take some time. If you are in China and run into connection issue, try export MXNET_GLUON_REPO=https://apache-mxnet.s3.cn-north-1.amazonaws.com.cn/ before starting the process to use the Gluon repo mirror in China

Thank you very much! However, when I load the Chinese word vector file, I am encountered the following problem, how should I solve it?

#words in training set: 15092
pretrained_file_name:  cc.zh.300-355cfcaf.npz
expected_file_hash:  355cfcafe71536226a1737aafb4530c9ba4fd09f
pretrained_file_path:  /home/.mxnet/embedding/fasttext/cc.zh.300-355cfcaf.npz
idx_to_token :  1999998 是
Traceback (most recent call last):
  File "dep_parser.py", line 297, in <module>
    parser.train(train_file='tests/biaffinedata/ctb.conll06', dev_file='tests/biaffinedata/ctb.conll06', test_file='tests/biaffinedata/ctb.conll06', save_dir='tests/biaffinedata/model', pretrained_embeddings=('cc.zh.300-355cfcaf.npz', 'embeddings'))
  File "dep_parser.py", line 133, in train
    min_occur_count)
  File "/home/workspace/biaffineparser-gluonnlp/parsing/parser/common/data.py", line 154, in __init__
    self._add_pret_words(pret_embeddings)
  File "/home/workspace/biaffineparser-gluonnlp/parsing/parser/common/data.py", line 185, in _add_pret_words
    pret_embeddings = gluonnlp.embedding.FastText('cc.zh.300')
  File "/home/anaconda3/lib/python3.6/site-packages/gluonnlp/embedding/token_embedding.py", line 979, in __init__
    self._load_embedding(pretrained_file_path, elem_delim=' ')
  File "/home/anaconda3/lib/python3.6/site-packages/gluonnlp/embedding/token_embedding.py", line 256, in _load_embedding
    pretrained_file_path=pretrained_file_path)
  File "/home/anaconda3/lib/python3.6/site-packages/gluonnlp/embedding/token_embedding.py", line 341, in _load_embedding_serialized
    deserialized_embedding = TokenEmbedding.deserialize(pretrained_file_path)
  File "/home/anaconda3/lib/python3.6/site-packages/gluonnlp/embedding/token_embedding.py", line 793, in deserialize
    idx_to_vec = nd.array(npz_dict['idx_to_vec'])
  File "/home/anaconda3/lib/python3.6/site-packages/numpy/lib/npyio.py", line 233, in __getitem__
    pickle_kwargs=self.pickle_kwargs)
  File "/home/anaconda3/lib/python3.6/site-packages/numpy/lib/format.py", line 664, in read_array
    array = numpy.ndarray(count, dtype=dtype)
MemoryError

[szha]

Hmm looks like the platform may have run out of memory. Could you verify?

[szha]

cc @leezu

[leezu]

If your system has limited memory, please disable out of vocabulary words handling: gluonnlp.embedding.FastText('wiki.zh', load_ngrams=False) instead of gluonnlp.embedding.FastText('wiki.zh', load_ngrams=True)

[leezu]

Feel free to reopen if you have further problems. Thanks