Skip to content

Tokenizer POS-tagger and Dependency-parser for Classical Chinese

License

Notifications You must be signed in to change notification settings

KoichiYasuoka/GuwenCOMBO

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Current PyPI packages

GuwenCOMBO

Tokenizer, POS-Tagger, and Dependency-Parser for Classical Chinese Texts (漢文/文言文), working with COMBO-pytorch.

Basic usage

>>> import guwencombo
>>> lzh=guwencombo.load()
>>> s=lzh("不入虎穴不得虎子")
>>> print(s)
# text = 不入虎穴不得虎子
1			ADV	v,副詞,否定,無界	Polarity=Neg	2	advmod	_	Gloss=not|SpaceAfter=No
2			VERB	v,動詞,行為,移動	_	0	root	_	Gloss=enter|SpaceAfter=No
3			NOUN	n,名詞,主体,動物	_	4	nmod	_	Gloss=tiger|SpaceAfter=No
4			NOUN	n,名詞,固定物,地形	Case=Loc	2	obj	_	Gloss=cave|SpaceAfter=No
5			ADV	v,副詞,否定,無界	Polarity=Neg	6	advmod	_	Gloss=not|SpaceAfter=No
6			VERB	v,動詞,行為,得失	_	2	parataxis	_	Gloss=get|SpaceAfter=No
7			NOUN	n,名詞,主体,動物	_	8	nmod	_	Gloss=tiger|SpaceAfter=No
8			NOUN	n,名詞,,関係	_	6	obj	_	Gloss=child|SpaceAfter=No

>>> t=s[1]
>>> print(t.id,t.form,t.lemma,t.upos,t.xpos,t.feats,t.head.id,t.deprel,t.deps,t.misc)
1   ADV v,副詞,否定,無界 Polarity=Neg 2 advmod _ Gloss=not|SpaceAfter=No

>>> print(s.to_tree())
 <════╗   advmod
 ═══╗═╝═╗ root
 <╗ ║   ║ nmod
 ═╝<╝   ║ obj
 <════╗ ║ advmod
 ═══╗═╝<parataxis
 <╗ ║     nmod
 ═╝<obj

>>> f=open("trial.svg","w")
>>> f.write(s.to_svg())
>>> f.close()

trial.svg guwencombo.load() has two options guwencombo.load(BERT="guwenbert-base",Danku=False). With the option BERT="guwenbert-large" the pipeline utilizes GuwenBERT-large. With the option Danku=True the pipeline tries to segment sentences automatically. to_tree() and to_svg() are borrowed from those of UD-Kanbun.

Kundoku usage

>>> import guwencombo
>>> lzh=guwencombo.load()
>>> s=lzh("不入虎穴不得虎子")
>>> t=guwencombo.translate(s)
>>> print(t)
# text = 虎の穴に入らずして虎の子を得ず
1			NOUN	n,名詞,主体,動物	_	3	nmod	_	Gloss=tiger|SpaceAfter=No
2		_	ADP	_	_	1	case	_	SpaceAfter=No
3			NOUN	n,名詞,固定物,地形	Case=Loc	5	obj	_	Gloss=cave|SpaceAfter=No
4		_	ADP	_	_	3	case	_	SpaceAfter=No
5	入ら		VERB	v,動詞,行為,移動	_	0	root	_	Gloss=enter|SpaceAfter=No
6	ずして		AUX	v,副詞,否定,無界	Polarity=Neg	5	advmod	_	Gloss=not|SpaceAfter=No
7			NOUN	n,名詞,主体,動物	_	9	nmod	_	Gloss=tiger|SpaceAfter=No
8		_	ADP	_	_	7	case	_	SpaceAfter=No
9			NOUN	n,名詞,,関係	_	11	obj	_	Gloss=child|SpaceAfter=No
10		_	ADP	_	_	9	case	_	SpaceAfter=No
11			VERB	v,動詞,行為,得失	_	5	parataxis	_	Gloss=get|SpaceAfter=No
12			AUX	v,副詞,否定,無界	Polarity=Neg	11	advmod	_	Gloss=not|SpaceAfter=No

>>> print(t.sentence())
虎の穴に入らずして虎の子を得ず

>>> print(s.kaeriten())
虎穴虎子>>> print(t.to_tree())
 ═╗<nmod(体言による連体修飾語)
 <╝ ║     case(格表示)
 ═╗═╝<obj(目的語)
 <╝   ║   case(格表示)
 ═╗═══╝═╗ root()
  ║     ║
 <╝     ║ advmod(連用修飾語)
 ═╗<╗   ║ nmod(体言による連体修飾語)
 <╝ ║   ║ case(格表示)
 ═╗═╝<╗ ║ obj(目的語)
 <╝   ║ ║ case(格表示)
 ═╗═══╝<parataxis(隣接表現)
 <advmod(連用修飾語)

translate() and reorder() are borrowed from those of UD-Kundoku.

Installation for Linux

pip3 install guwencombo

Installation for Cygwin64

Make sure to get python37-devel python37-pip python37-cython python37-numpy python37-cffi gcc-g++ mingw64-x86_64-gcc-g++ gcc-fortran git curl make cmake libopenblas liblapack-devel libhdf5-devel libfreetype-devel libuv-devel packages, and then:

curl -L https://raw.githubusercontent.com/KoichiYasuoka/UniDic-COMBO/master/cygwin64.sh | sh
pip3.7 install guwencombo

Installation for macOS

g++ --version
pip3 install guwencombo --user
python3 -m spacy download en_core_web_sm --user

If you fail to install Jsonnet, try below before installing GuwenCOMBO:

( echo '#! /bin/sh' ; echo 'exec gcc `echo $* | sed "s/-arch [^ ]*//g"`' ) > /tmp/clang
chmod 755 /tmp/clang
env PATH="/tmp:$PATH" pip3 install jsonnet --user

If you fail to install fugashi, try to install MeCab before installing GuwenCOMBO:

cd /tmp
git clone --depth=1 https://github.com/taku910/mecab
cd mecab/mecab
./configure --with-charset=UTF8
make && sudo make install

Reference

About

Tokenizer POS-tagger and Dependency-parser for Classical Chinese

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published