LexRank Performance when corpus is large

[MonkandMonkey]

My corpus contains 300 paragraphs, and the speed is slow. More than 30 mins.
Could you please introduce sumy's performance ? And which stage will make it slow when corpus is large.
Thanks!

[miso-belica]

Hi, can you share the corpus and let me know exact command that is slow?

[MonkandMonkey]

I am so sorry the corpus can't be shared due to its privacy. Here is the description of the corpus.

• lang: Chinese (I replace the default jieba tokenizer with ltp tokenizer to cut words.)
• Text format:

1. Each line is a doc, and it may contains several sentences.
2. Doc len varies from 10 to 2000+. (Because our corpus contains 'titles' and 'contents'.)
3. Some sentences may be very long.

• Here is the code:

from sumy.parsers.plaintext import PlaintextParser
from sumy.nlp.tokenizers import Tokenizer
from sumy.summarizers.lex_rank import LexRankSummarizer as Summarizer
from sumy.nlp.stemmers import Stemmer
from sumy.utils import get_stop_words

def summary_docs(self, src_file, output_file):
        Tokenizer.SPECIAL_WORD_TOKENIZERS[self.lang] = LTPTokenizer()
        parser = PlaintextParser.from_file(src_file, Tokenizer(self.lang))
        stemmer = Stemmer(self.lang)

        summarizer = Summarizer(stemmer)
        summarizer.stop_words = get_stop_words(self.lang)

        with open(output_file, "w", encoding=Config.default_encoding) as fw:
            logging.debug(parser.document.paragraphs)
            for sentence in summarizer(parser.document, self.summary_sentence_cnt):
                fw.write(str(sentence) + "\n")
        logging.info("finish summarization, summary is saved in: {}".format(Config.summary_dir))

Thank you very much.