Exception has occurred: AttributeError // module 'pytextrank' has no attribute 'TextRank'

[isabel-martin-pineiro]

Hi, there:

I am trying to implement the following code, as part of a large class called "Analysis". This class includes several large-texts analysis, as modeling text or key phrase extraction.

I get the error "Exception has occurred: AttributeError // module 'pytextrank' has no attribute 'TextRank'". Is there any newer method available?

def extract_topwords_pytextrank(self):
        for newspaper in self.press_list:
            for article_id in self.analysis_kw_dict[self.keyword_1][newspaper]["articles"]:
                cadena_title = ""
                cadena_subtitle = ""
                cadena_body = ""
                cadena_title += self.analysis_kw_dict[self.keyword_1][newspaper]["articles"][article_id]["title"]
                cadena_subtitle += self.analysis_kw_dict[self.keyword_1][newspaper]["articles"][article_id]["subtitle"]
                cadena_body += self.analysis_kw_dict[self.keyword_1][newspaper]["articles"][article_id]["body"]
                lista_cadenas = [cadena_title, cadena_subtitle, cadena_body]
                for string in lista_cadenas:
                    tr = pytextrank.TextRank()
                    nlp = tr.parser()  # We create an analyzer
                    doc = nlp(cadena)     # We process text
                    # stopwords in Spanish
                    stop_words = set(pytextrank.utils.importance.stop_words("es"))
                    # Grammatical categories that are considered keywords 
                    allowed_pos_tags = ["NN", "NNS", "NNP", "NNPS", "JJ"]
                    # We get keywords using PyTextRank
                    keywords = [word for word in doc._.phrases
                                if word.lower() not in stop_words and
                                any(tag in allowed_pos_tags for _, tag in word.tags)]
                    # We print keywords
                    for keyword in keywords:
                        print(keyword.cadena)

[ceteri]

Hi @isabel-martin-pineiro ,

The way to use a spaCy pipeline is different. In the code shown above, the tr = pytextrank.TextRank() kind of pattern is not supported by spaCy pipelines.

If I understand your use case correctly, instead a typical use would be more like:

import spacy
import pytextrank

# load the language model needed
nlp = spacy.load("es_core_news_sm")  # or replace with another for `es`

# configure the stop words
# see:
# https://derwen.ai/docs/ptr/sample/#stop-words
# https://derwen.ai/docs/ptr/ref/#__init__-method
es_stopwords = "path/to/stop/words/file"  # replace with file path

nlp.add_pipe(
    "textrank",
    config = {
        "stopwords": es_stopwords,
    },
)

# grammatical categories which are considered keywords 
allowed_pos_tags = ["NN", "NNS", "NNP", "NNPS", "JJ"]

lista_cadenas = ...  # set up the data structures for your corpus

for cadena in lista_cadenas:
    doc = nlp(cadena)

    keyphrase = [
        word for word in doc._.phrases
        if any(tag in allowed_pos_tags for _, tag in word.tags)
    ]

    for phrase in keyphrase:
        print(phrase)

Does that work for what you needed?

Note: depending on the language, you may need to adjust the list of pos_kept tags. If this list is filtered too much, it can break how a textgraph algorithm works internally.

[isabel-martin-pineiro]

Thanks a lot!
One doubt: where do we get a stopword file in Spanish? Do we need to create one?

[ceteri]

Great, I'm glad that helped :)

We don't have a defined stopwords list in Spanish
( perro, deberíamos tener uno, porque trabajo mucho en Madrid, Valencia, A Coruña,... :)

Usually the stopwords depend on each application in any case.

There is another collection of common stopwords, in many languages -- if you want to adapt this?
https://github.com/stopwords-iso/stopwords-es