Version 1.0 (Moving Tablesaw to Java 11)

[lwhite1]

I decided to update this discussion thread as a way to keep people informed on the plans for Tablesaw development in the coming months.

There are now two ongoing branches: The java-8 branch, currently at version 0.43.2-SNAPSHOT will stay on Java 8 and will get some non-breaking enhancements (including backports, mostly provided by the community) and bug-fixes.

The master branch, currently at 1.0.0-SNAPSHOT, has moved to Java 11, a long-term support version. This branch will have breaking changes when it is released, and major development going forward will be on this branch. The current LTS version of Java is 17, with 18 as the most recent branch. At some point I will decide whether to release 1.0 on Java 11, or choose one of the more recent Java version. A list of important changes on this branch will be maintained at the bottom of this post.

For Tablesaw to be included in projects that use the Java module system, as real modules, we need to be on Java 9.0 or later as the language level. The current fallback is to put Tablesaw on the class path rather than the module path for the modules. As far as I know, this works ok, but it breaks module encapsulation. Some projects we use do not support modules, making it difficult or impossible to offer full module support in the short-term, but we can be ready to offer that support when they're ready.

I don't yet know how this will play with our sub-projects, especially those that implement IO features, which use reflection. It may require changes to the API to remove this requirement.

The CI system has been updated to use Java 11 and 17 for platform builds, so updates to the java-8 branch will need to be tested locally.

Changes coming on version 1.0

Here are some of the changes that are currently on the 1.0 branch:

1. Update the language level to Java 11 (or higher)
2. Change the poms to no longer build shaded-jars. Shaded jars break module encapsulation
3. Remove all deprecated code that is currently in the java-8 branch. Most of this has been deprecated for years, so hopefully it is not in wide use.
4. Extract jsplot module into its own GitHub project, after eliminating the dependencies on Tablesaw core. A revised jsplot module will provide an interface to the new project.
5. Added support for sort-merge join as an alternative to the current "nested-loop" or cross product join to improve performance where both Tables are large. Performance when both tables are the same size can approach O(n^2) where at least one column has low-cardinality. The new algorithm is roughly 100X faster on the worst case scenarios. Implemented an initial pass to having the algorithm selected at runtime based on the nature of the tables being joined. This should provide more consistently acceptable join performance.
6. Subsume TextColumn into StringColumn to eliminate the usability issues caused by having multiple column types for text data
7. Adding IO support for Apache Arrow.
8. Minor performance enhancements for BooleanColumn.

Candidate changes for version 1.0

These changes are under development, but not yet merged into master.

• Implement function to split a StringColumn into multiple StringColumns using some provided delimiter.
• Decide whether to keep Saw file format (and remove the @beta annotation) or remove it in favor of the arrow format.

[hmf]

Just one note: your library is also used by JVM compatible languages. In my case Scala, but their are others such as Kotlin, Goovy and Clojure. In the case of Scala it is much easier to use the class path only. Its not that we cannot use modules, but adding the "--add-opens", "--add-modules" and "--add-exports" to the command line arguments is a pain. So I would say that one should try to avoid the need for these ( "--module-path" and "--add-modules" will always be needed).

Having said this, it should also be possible to add the libraries as unnamed modules when used in the classpath. This was designed for backwards compatibility. Not all libraries allow this (for example OpenJFX). I think this will will work irrespective of wether or not JTablesaw uses modules or not. Maybe this can be added to some test?

HTHs

[lwhite1]

Thank you, @hmf . That's a good point. Sometimes I forget that people use Tablesaw from Kotlin and Scala.

TBH, I need to do a great deal more reading about the module system as I haven't used it before. I think at some point we have to take it on, since being stuck on Java 8 forever is very limiting and will cause more problems for projects going forward as more people make the switch.

[lwhite1]

One possible approach to moving forward - not ideal for sure - would be to fork the project and leave the current version in bug-fix-only mode, while doing future development on a path that stays up-to-date with the long-term support versions. Thoughts?

[aadrian]

would be to fork the project

@lwhite1
Why not just create a simple branch? (e.g. JAVA_8 branch) and create releases from there for the old 0.x.x ones (fixes or not) and bump the version number to 1.x for the master with the java >= 11 development ?

This way one can decide at any point if just a bugfix release is needed, (or also some feature change for the old branch - if it's need and the community provies the patch) ?

[lwhite1]

I thought about it yesterday and came to the same conclusion. That way, I could also put the breaking change unifying StringColumn in the 1.0 branch only, which makes sense.