John Snow Labs Spark-NLP 3.1.2: New and improved XLNet with support for external Transformers, better documentations, bug fixes, and other improvements!

[maziyarpanahi]

---

Overview

We are pleased to release Spark NLP 🚀 3.1.2! We have a new and much-improved XLNet annotator with support for HuggingFace 🤗 models in Spark NLP. We managed to make XlnetEmbeddings almost 5x times faster on GPU compare to prior releases!

As always, we would like to thank our community for their feedback, questions, and feature requests.

---

New Features

• Migrate XlnetEmbeddings to TensorFlow v2. This allows the importing of HuggingFace XLNet models to Spark NLP
• Migrate XlnetEmbeddings to BatchAnnotate to allow better performance on accelerated hardware such as GPU
• Dynamically extract special tokens from SentencePiece model in XlmRoBertaEmbeddings
• Add setIncludeAllConfidenceScores param in NerDLModel to merge confidence scores per label to only predicted label
• Fully updated Annotators page with full examples in Python and Scala
• Fully update Transformers page for all the transformers in Spark NLP

---

Bug Fixes & Enhancements

• Fix issue with SymmetricDeleteModel
• Fix issue with encoding unknown bytes in RoBertaEmbeddings
• Fix issue with multi-lingual UniversalSentenceEncoder models
• Sync params between Python and Scala for ContextSpellChecker
  • change setWordMaxDist to setWordMaxDistance in Scala
  • change setLMClasses to setLanguageModelClasses in Scala
  • change setWordMaxDist to setWordMaxDistance in Scala
  • change setBlackListMinFreq to setCompoundCount in Scala
  • change setClassThreshold to setClassCount in Scala
  • change setWeights to setWeightedDistPath in Scala
  • change setInitialBatchSize to setBatchSize in Python
• Sync params between Python and Scala for ViveknSentimentApproach
  • change setCorpusPrune to setPruneCorpus in Scala
• Sync params between Python and Scala for RegexMatcher
  • change setRules to setExternalRules in Scala
• Sync params between Python and Scala for WordSegmenterApproach
  • change setPosCol to setPosColumn
  • change setIterations to setNIterations
• Sync params between Python and Scala for ViveknSentimentApproach
  • change setCorpusPrune to setPruneCorpus
• Sync params between Python and Scala for PerceptronApproach
  • change setPosCol to setPosColumn
• Fix typos in docs: Fix typo "lanuch" -> "launch" #5766 and Add missing code block end #5775 thanks to @brollb

---

Performance Improvements

Introducing a new batch annotation technique implemented in Spark NLP 3.1.2 for XlnetEmbeddings annotator to radically improve prediction/inferencing performance. From now on the batchSize for these annotators means the number of rows that can be fed into the models for prediction instead of sentences per row. You can control the throughput when you are on accelerated hardware such as GPU to fully utilize it.

---

Backward compatibility

We have migrated XlnetEmbeddings to TensorFlow v2, the earlier models prior to 3.1.2 won't work after this release.
We have already updated the models and uploaded them on Models Hub. You can use pretrained() that takes care of it automatically or please make sure you download the new models manually.

---

Documentation

• HuggingFace to Spark NLP
• Models Hub with new models
• Spark NLP publications
• Spark NLP in Action
• Spark NLP documentation
• Spark NLP Workshop notebooks
• Spark NLP training certification notebooks for Google Colab and Databricks
• Spark NLP Display for visualization of different types of annotations
• Discussions Engage with other community members, share ideas, and show off how you use Spark NLP!

---

Installation

Python

#PyPI

pip install spark-nlp==3.1.2

Spark Packages

spark-nlp on Apache Spark 3.0.x and 3.1.x (Scala 2.12 only):

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp_2.12:3.1.2

pyspark --packages com.johnsnowlabs.nlp:spark-nlp_2.12:3.1.2

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:3.1.2

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu_2.12:3.1.2

spark-nlp on Apache Spark 2.4.x (Scala 2.11 only):

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-spark24_2.11:3.1.2

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-spark24_2.11:3.1.2

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-gpu-spark24_2.11:3.1.2

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu-spark24_2.11:3.1.2

spark-nlp on Apache Spark 2.3.x (Scala 2.11 only):

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-spark23_2.11:3.1.2

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-spark23_2.11:3.1.2

GPU

spark-shell --packages com.johnsnowlabs.nlp:spark-nlp-spark23-gpu_2.11:3.1.2

pyspark --packages com.johnsnowlabs.nlp:spark-nlp-gpu-spark23_2.11:3.1.2

Maven

spark-nlp on Apache Spark 3.0.x and 3.1.x:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp_2.12</artifactId>
    <version>3.1.2</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu_2.12</artifactId>
    <version>3.1.2</version>
</dependency>

spark-nlp on Apache Spark 2.4.x:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-spark24_2.11</artifactId>
    <version>3.1.2</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu-spark24_2.11</artifactId>
    <version>3.1.2</version>
</dependency>

spark-nlp on Apache Spark 2.3.x:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-spark23_2.11</artifactId>
    <version>3.1.2</version>
</dependency>

spark-nlp-gpu:

<dependency>
    <groupId>com.johnsnowlabs.nlp</groupId>
    <artifactId>spark-nlp-gpu-spark23_2.11</artifactId>
    <version>3.1.2</version>
</dependency>

FAT JARs

• CPU on Apache Spark 3.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-assembly-3.1.2.jar

• GPU on Apache Spark 3.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-assembly-3.1.2.jar

• CPU on Apache Spark 2.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-spark24-assembly-3.1.2.jar

• GPU on Apache Spark 2.4.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-spark24-assembly-3.1.2.jar

• CPU on Apache Spark 2.3.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-spark23-assembly-3.1.2.jar

• GPU on Apache Spark 2.3.x: https://s3.amazonaws.com/auxdata.johnsnowlabs.com/public/jars/spark-nlp-gpu-spark23-assembly-3.1.2.jar

---

This discussion was created from the release John Snow Labs Spark-NLP 3.1.2: New and improved XLNet with support for external Transformers, better documentations, bug fixes, and other improvements!.