👩‍💻 API next steps: checklist

[jbesomi]

The following contains a high-level view of what will be the next main enhancement steps. This document will be kept up-to-date and improved frequently. This work will be mainly conducted by @mk2510 and @henrifroese as part of their SummerOfCode project.

---

1. Version 1.10

   • Every representation function to receive as input a TokenSeries TokenSeries as input to every representation function #44
   • Decouple TF-IDF L2-normalization and TF-IDF tfidf(s): remove normalization, improve docstring #76
   • Rename term_frequency to count() + add functionterm_frequency count(s) and term_frequency(s) #61
   • Introduce HeroSeries
   • Add ~ hero.norm(RepresetationSeries, "l1"/"l2")
   • Can we avoid the use of VectorSeries/TokenSeries?
   • All representation functions to deal with HeroSeries + (DocumentTermDF) Support "Pandas Series Representation" #43
   • Update README + getting-started.md
   • Push a new version to PyPi

2. Performance: speed-up the library

   • Most of Texthero data structure are list of list ([["a", "document"], ["another", "document"]]), can we leverage parallelization? We can learn from spaCy. Mandatory read: 100-times-faster-nlp; look at this for parallelization
   • Make spaCy function faster + Dask vs Spacy Make spaCy-nlp functions faster #65
   • Depending on the previous task, evaluate if we want to have as default tokenizer spaCy: tokenize with Spacy #131

3. Software development:

   • Integrate checking for correct Series types (Kind of Pandas Series #60, Check if Series consists of strings only, instead of casting to unicode #55, ...)
   • Check hero functions work with np.nan All function to deal with np.nan #86

4. Support Embeddings through Flair

   • Add hero.embed(s, flairEmbedding)

5. Add Topic Modeling

   • Add topic modeling support under representation Implement/support/explain topic modelling #42
     This include also "topic modeling visualization" to get insights out of it
   • Add a blog article on how topic modeling with Texthero works

6. Extra

   1. test coverage
   2. expand multilingual: more languages; recognize languages and select correct one
   3. (low priority) Text summarization (Preprocessing: explain how to create a custom pipeline #38) and characteristic terms (Characteristic Terms & Keywords #2)

[henrifroese]

Merge-Plan

• Change representation_series to DataFrame #156 (representation series to multicolumn). Branched from Texthero Master
• HeroTypes in Representation; DataFrame in _types #157 (hero types in representation & DocumentTermDF in _types). Branched from Change representation_series to DataFrame #156
  Add pandas __setitem__ support for DocumentTermDF #158 (add pandas setitem support for DocumentTermDF). Branched from Change representation_series to DataFrame #156
• Fix type checks to not fail if first value in series is NaN/missing. #174 (Fix type checks). Branched from Texthero Master
• Update README.md #117 (getting-started), Update getting_started.md #118 (README), Getting started: Kind of Series (HeroSeries) #135 (getting-started hero-types). Branched from Texthero Master
• RELEASE NEW VERSION
• Support for Flair Embeddings: hero.embed(s, flair_embedding) function #146 (Flair Embeddings). Branched from Texthero Master
• Unit-test/Travis CI: Cleaner Log, Features; Closes #137 #160 (Travis Clean-Up). Branched from Texthero Master
• Added Pre commit hook #161 (Pre-commit hook). Branched from Texthero Master
• Speed-Up Preprocessing + NLP #162 (Speed-Up Preprocessing+NLP). Branched from Texthero Master
• Topic Modelling and Visualization #163 (Topic Modelling w/ Visualizations). Branched from Texthero Master
• Fix Term_Frequency #165 (Fix term_frequency). Branched from Texthero Master
• Implement train_test_split #167 (Train-Test Split). Branched from Texthero Master
• Implement hero.describe(s), Closes #166 #168 (Describe DF). Branched from Texthero Master
• Implement filter_extremes #169 (filter extremes). Branched from Texthero Master
• Add ClusterSeries to Hero Series Types #170 (ClusterSeries Type). Branched from Texthero Master
• Add Visualisation Tutorial #175 (Visualization Tutorial). Branched from Texthero Master
• Add NLP Tutorial #176 (NLP Tutorial). Branched from Texthero Master
• Implement show_dataframe #177 (Show DataFrame). Branched from Texthero Master
• Visualize hero.describe with plots #178 (Visualize Describe DF). Branched from Implement hero.describe(s), Closes #166 #168