Version 2.0 Release Planning

[henrifroese]

Hi, so this is meant as a general discussion about the new release. Of course, there's always a few bugs to fix and small changes to be made - that can be done at any time. However, for big changes, we want to do separate releases that are well-documented and update the pip repository and the website.

There have been 100 commits to the master branch since the last release, some with small bugfixes and some with big changes to the public API, so this will be a big release and we have to take care of lots of stuff.

I believe the first three things we should discuss are:

1. Which open PRs that modify the public API or make significant internal changes we still want to have included in the release,
2. What new features that are not in any PR yet we want to include,
3. What updates to documentation we want to include.

As a start, here is my opinion on the three:

1. Opinion on open PRs to include / not include in the new release

I'm basing my opinion on discussions with @jbesomi that we should focus on clean, well-documented, tested, and especially really useful (for many users) new features that do not complicate the library too much.

(I'm using green circles to say "should be included", yellow for "maybe", red for "no")

• 🟢 Flair Embeddings (Support for Flair Embeddings: hero.embed(s, flair_embedding) function #146 ): I think this should definitely be included as it greatly increases functionality and should be very useful for many users, while still being well-integrated into the library. There's a little work that still needs to be done there though. I added a small todo-list in the PR.
• 🟡 Parallelize Preprocessing + NLP (Speed-Up Preprocessing + NLP #162 ): This is a rather big change (although most of the new lines are tests 🍔 ). It speeds up many preprocessing and nlp functions as much as possible, is well-documented and well-tested. However, it makes it a little more challenging for new developers to contribute to those modules (of course, they can just write non-parallelized functions as before, but the parallelized ones might be confusing). Thus, I think that this should be documented a little clearer in the code for developers, and it should be included in the getting-started how users can change the thresholds for parallelization; then I think this can be included. I added those two todos to the PR.
• 🔴 Show_Dataframe (make dataframe searchable with HTML datatable) (Implement show_dataframe #177 ): I believe we already discussed that in its current state, this should not be included as it is not really polished yet. I think it should remain open and would be a great feature if it was usable for bigger datasets, better tested and nicer to look at. I added those as todos to the PR.
• 🟢 Hero.Describe (Implement hero.describe(s), Closes #166 #168 ): This is a very lightweight function that I think will be immensely useful for many users. I can see many using this as the first function for every new dataset, and it's well-tested and easy-to-use. Thus, I think it should definitely be included.
• 🟡 Visualize hero.describe (Visualize hero.describe with plots #178 ): This just visualizes hero.describe using plotly, which is already a dependency anyways. I think that this is a nice function that separates Texthero from other NLP libraries as it creates a nice, useful visualization in just one line of code that tells users a lot about their data. However, compared to hero.describe, the only thing this visualization really adds is the exact distribution of document lengths in the bottom left plot, instead of just percentiles in the output of hero.describe. Still, I think this is a low-overhead visualization that's nicer to look at than the raw output of hero.describe
• 🟢 ClusterSeries (Add ClusterSeries to Hero Series Types #170): Just had another look, seems like a straightforward PR that will be useful going ahead for users.
• 🟢 filter_extremes (Implement filter_extremes #169 ): This is one of my favorite additions that I'd like to see in the library. It's intuitive, documentation is clear, and it's really useful for users.
• 🟡 Topic Modelling and Visualization (Topic Modelling and Visualization #163 ): This PR is only useful for users if it is accompanied by lots of documentation & a separate tutorial in my opinion. It should definitely make its way into the library at some point though as it's extremely useful for users, if it is explained well. Thus, I think that either
  • (a) We split this into separate PRs, go through the review process again, add a tutorial, and feature it in the release
  • (b) Same as (a), but we finish the release first, and then include it in another separate release with lots of tutorials etc.

2. Opinion on new features that should still make it into the new release

Not sure I have anything here. I think @jbesomi you wanted to make tokenization a mandatory "first step", maybe you can expand on that?

3. Opinion on what updates to documentation we want to include

I think it makes sense to include all documentation & tutorials that were written until now. It's probably easiest if we first decide on 1. and 2. and then go through, update and merge #117 , #118 , #135 , #175 , #176 , #183 to make sure all new features are explained and mentioned. Then, we definitely still need a separate representation tutorial as far as I can see. Finally, we need release notes!

[jbesomi]

Hi Henri,

That's great, thank you. I agree with all points you mentioned before and with the button colors. For now, let's focus on the green ones, and eventually, we will also integrate the yellow ones in the next release.

Before the next release, what's still needs to be done is to finalize the documentation and that's where we should invest more of our effort. The idea is to have very simple and polished documentation that takes a maximum of 20 minutes to be completely read. We want users to quickly understand how the library work.

...

[henrifroese]

Next step: Merging the green points from above. First: #168

@jbesomi Just had another look at this, looks good and useful. I'll make a quick update to add docstring parameter description and make stuff more clear, but then this can be merged as a first step.

Edit: just did, can be merged from my side now.

[jbesomi]

That's good! I'm out right now and will go back to work on Tuesday. At this moment, I will look into it and merge it. Thanks 🎉

[henrifroese]

@jbesomi There are issues with dependencies etc leading to failing installs. They will be investigated in #206 , so we shouldn't merge anything until that's fixed

[jbesomi]

Ok, thanks! I will look into #206.

[henrifroese]

@jbesomi just merged the updated master into #168 so the checks are now passing there and it can be merged