How does scattertext generate set of words for plotting, and compute x and y coordinates?

[ebaggott]

Hello! I have two questions:

1) How does produce_scattertext_explorer() come up with the set of words to plot?

I've encountered what I think is a discrepancy: After I've set the minimum term frequencies to zero, and the pmi threshold coefficient to zero, produce_scattertext_explorer() generates a set of words that is smaller than the set of words in the corpus TDF.

html = st.produce_scattertext_explorer(corpus,
category='pre',
category_name='China',
not_category_name='Russia',
minimum_term_frequency=0,
minimum_not_category_term_frequency=0,
pmi_threshold_coefficient=0,
return_data=True)

mydata = [t for t in html.items()][1][1]
myterms = [row['term'] for row in mydata]
tdf = corpus.get_term_freq_df()
terms = [t for t in tdf.index]

len(myterms) # 157540
len(terms) # 164689 ... why are these different?

2) How does produce_scattertext_explorer() calculate x and y coordinates?

Would it be possible to add this to the documentation? That would be really, really helpful.

Many thanks, really appreciate your help!

[jonessarae]

I also have the same question.

I'm trying to plot over 800 topics. However around 600 topics actually show up in the plot, and the topic I was looking for didn't show up.

Here's my code:

topic_feature_builder = st.FeatsFromTopicModel(topic_model)

corpus =(st.CorpusFromParsedDocuments(df,
category_col="Awarded",
parsed_col="parsed_combined",
feats_from_spacy_doc=topic_feature_builder)
.build().remove_terms(stopwords, ignore_absences=True))

len(topic_feature_builder.get_top_model_term_lists()) # 819 topics
len(list(corpus.get_metadata_doc_count_df().index)) #626 topics in corpus

Is there a way to add the missing topics to the corpus?

[JasonKessler]

Thanks for the bug report.

I appreciate the code sample you've provided, but it's hard to provide support if I can't recreate the issue on my own. Could you make a colab notebook recreating the problem, or post code which can be run independently which reproduces the issue?

My initial sense is that some of the topics in may not appear in your data. This could be due to differences in tokenization between the topic model and spaCy's tokenization or simply because that no words from certain topics occurred in your dataset.

[JasonKessler]

I looked at @jonessarae's data and custom topic model, and it looks like the issue was that FeatsFromTopicModel wouldn't work with topic models which had multi-token entries. I added a PhraseFeatsFromTopicModel class which will find exact-match occurrences of entries in topic models as part of version 0.0.2.67.