Support language which need tokenizer (Chinese, Japanese .etc)

[eromoe]

I think iepy need a common interface to embed a tokenizer to support language like Chinese, Japanese .etc.

There is a old ie project with gui named GATE, it contain a pre-trained model and dataset, maybe helpful
https://gate.ac.uk/sale/tao/splitch15.html#sec:misc-creole:language-plugins:chinese

[francolq]

Hello. The preprocessing pipeline can be customized to introduce a different tokenizer. See for instance:

https://github.com/awolfmann/PLN-2015/blob/practico4/information_extraction/resoluciones-unc/bin/preprocess.py

[eromoe]

Hello @francolq ,
I have seen how to customise in docs:

    pipeline = PreProcessPipeline([
        CustomTokenizer(),
        CustomSentencer(),
        CustomLemmatizer(),
        CustomPOSTagger(),
        CustomNER(),
        CustomSegmenter(),
    ], docs)
    pipeline.process_everything()

Then I look into the code , preprocess.tokenizer.TokenizeSentencerRunner seems not be used in anywhere. And I found:

• one pipeline may have multiple runner
• one runner may have step or not

As I see, there is not just as simple as adding a tokenizer since some runners are relative.It is a little hard to customise without knowing the input and output of each runner and step format and the runner api design principle (Currently I have to view the code and tried to understand what it does, but due to knowledge and language limitation, I may stuck at some place). I would like to help to make iepy compatible with CJK language if anyone could provide the api principle to write the runners. @machinalis @jmansilla

[jmansilla]

Sorry the delay respect this talk. Can I still help here @eromoe ?

[YanWenqiang]

@eromoe Right now, I want iepy to customize to Chinese, could you give me a hand ?

[eromoe]

@YanWenqiang Sorry, I was just need the annotator and object binding of iepy, since it was not easy to integrate Chinese , I have already made my own now.

[YanWenqiang]

@eromoe All right. Thanks a lot. Now I was also met with this trouble, I really need someone could help me.

[hwaking]

@eromoe I am doing Chinese EMR information extraction ， can i use iepy to do entity relationship extraction ？