Module 3 Advanced NLP with Spark NLP.ipynb

{"cells":[{"cell_type":"markdown","source":["In this module, we'll explore Named Entity Recognition (NER) and Sentiment Analysis using Spark NLP with the Amazon reviews dataset. We'll identify the most common named entities mentioned in the reviews and analyze the overall sentiment of those reviews."],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{},"nuid":"5a2d4b82-9d14-4969-a4e4-4da673ee4dd8","inputWidgets":{},"title":""}}},{"cell_type":"markdown","source":["Import the necessary modules from PySpark and Spark NLP. These include functions and types from PySpark, and various annotators and base classes from Spark NLP."],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{},"nuid":"c8b17c74-8c7b-48a1-92fc-4abc97b349b3","inputWidgets":{},"title":""}}},{"cell_type":"code","source":["from pyspark.ml import Pipeline\nfrom pyspark.sql.functions import col, udf\nfrom pyspark.sql.types import StringType\n\nimport sparknlp\nfrom sparknlp.annotator import *\nfrom sparknlp.base import *"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{"rowLimit":10000,"byteLimit":2048000},"nuid":"6bf3277c-5208-4d71-aeaa-104ff9468c0f","inputWidgets":{},"title":""}},"outputs":[],"execution_count":0},{"cell_type":"code","source":["text_data_path = \"dbfs:/databricks-datasets/amazon/data20K\"\ntext_df = spark.read.parquet(text_data_path, header=True, inferSchema=True)"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{"rowLimit":10000,"byteLimit":2048000},"nuid":"09cba179-d734-44a7-9e57-20ea843d10d4","inputWidgets":{},"title":""}},"outputs":[],"execution_count":0},{"cell_type":"markdown","source":["### Named Entity Recognition\n\nOnce our data is loaded, we're ready to start processing it. For our first task, we're going to identify named entities in the reviews. Named entities are real-world objects such as persons, locations, organizations, and so on, that can be denoted with a proper name.\n\nNamed Entity Recognition (NER) is a subtask of information extraction that seeks to locate and classify named entities in text into predefined categories such as person names, organizations, locations, medical codes, time expressions, quantities, monetary values, percentages, etc.\n\nWe'll create a processing pipeline with Spark NLP to do this. A pipeline is a sequence of stages where each stage is either a Transformer or an Estimator. These stages are run in order, and the input DataFrame is transformed as it passes through each stage."],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{},"nuid":"29996eed-8e69-44de-b9db-b4fb50d12e3c","inputWidgets":{},"title":""}}},{"cell_type":"code","source":["# NER pipeline to identify and classify named entities in the reviews:\n\ndocument_assembler = DocumentAssembler() \\\n    .setInputCol(\"review\") \\\n    .setOutputCol(\"document\")\n\nsentence_detector = SentenceDetector() \\\n    .setInputCols([\"document\"]) \\\n    .setOutputCol(\"sentences\")\n\ntokenizer = Tokenizer() \\\n    .setInputCols([\"sentences\"]) \\\n    .setOutputCol(\"tokens\")\n\nembeddings = WordEmbeddingsModel.pretrained(\"glove_100d\") \\\n    .setInputCols([\"document\", \"tokens\"]) \\\n    .setOutputCol(\"embeddings\")\n\nner_model = NerDLModel.pretrained(\"ner_dl\", \"en\") \\\n    .setInputCols([\"sentences\", \"tokens\", \"embeddings\"]) \\\n    .setOutputCol(\"ner_tags\")\n\nner_converter = NerConverter() \\\n    .setInputCols([\"sentences\", \"tokens\", \"ner_tags\"]) \\\n    .setOutputCol(\"entities\")\n\npipeline = Pipeline(stages=[\n    document_assembler,\n    sentence_detector,\n    tokenizer,\n    embeddings,\n    ner_model,\n    ner_converter\n])"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{"rowLimit":10000,"byteLimit":2048000},"nuid":"d13c9bf0-d8bf-4ef2-83b4-ac6ee2eee2b1","inputWidgets":{},"title":""}},"outputs":[{"output_type":"stream","output_type":"stream","name":"stdout","text":["glove_100d download started this may take some time.\nApproximate size to download 145.3 MB\n\r[ | ]\r[OK!]\nner_dl download started this may take some time.\nApproximate size to download 13.6 MB\n\r[ | ]\r[OK!]\n"]}],"execution_count":0},{"cell_type":"markdown","source":["We're using the GloVe embeddings pretrained on 6 billion words (\"glove_100d\") to generate **word embeddings** which are then provided as input during named entity recognition. You can replace \"glove_100d\" with [any other embeddings](https://nlp.johnsnowlabs.com/models?type=model&q=glove) you prefer.\n\n<img src=\"https://www.tensorflow.org/static/text/guide/images/embedding.jpg\"  width=\"600\" height=\"300\">\n\nThe **NerDLModel** is a Named Entity Recognition model trained by a deep learning approach. It assigns to every word in a text a tag that signifies whether the word is a named entity or not, and what category it belongs to. It's trained using a variety of neural network architectures, such as Char CNNs - BiLSTM - CRF and Bert - BiLSTM - CRF.  \nIn the code above, we are loading a pretrained NER model (named \"ner_dl\") for English (\"en\"). We set the input columns to be \"sentences\" and \"tokens\". This means that the model will perform NER on the tokenized sentences from our text. The output of this stage is a new column, \"ner_tags\", which contains the NER tags assigned by the model to each word.\n\nThe **NerConverter** is used to convert the output from the NerDLModel into a more readable format. It groups together consecutive tokens with the same NER tag into single entities and classifies them according to their tag.  \nWe're setting the input columns to be \"sentences\", \"tokens\", and \"ner_tags\". This means that the NerConverter will take the tokenized sentences and their corresponding NER tags as input. The output of this stage is a new column, \"entities\", which contains the named entities extracted from the text. Each entity is represented as a chunk of text along with its NER tag."],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{},"nuid":"149e4510-209d-4bdb-9f42-0359c8e81dce","inputWidgets":{},"title":""}}},{"cell_type":"markdown","source":["We will now transform the reviews DataFrame using the NER pipeline:"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{},"nuid":"cbe591a9-ad13-42e2-83ed-302539d39b22","inputWidgets":{},"title":""}}},{"cell_type":"code","source":["pipeline_model = pipeline.fit(text_df)\nner_result_df = pipeline_model.transform(text_df)"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{"rowLimit":10000,"byteLimit":2048000},"nuid":"68edda99-4a9d-4186-8dbb-fe490cdacd1b","inputWidgets":{},"title":""}},"outputs":[],"execution_count":0},{"cell_type":"code","source":["ner_result_df.show(4)\n\n# ner_result_df.select(\"review\", \"entities\").show(2, truncate=False)"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{"rowLimit":10000,"byteLimit":2048000},"nuid":"88cd72ba-9bef-4db1-bf76-6545cd29d7cf","inputWidgets":{},"title":""}},"outputs":[{"output_type":"stream","output_type":"stream","name":"stdout","text":["+------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+\n|rating|              review|            document|           sentences|              tokens|          embeddings|            ner_tags|            entities|\n+------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+\n|   4.0|Worked as expecte...|[{document, 0, 10...|[{document, 0, 62...|[{token, 0, 5, Wo...|[{word_embeddings...|[{named_entity, 0...|                  []|\n|   5.0|This mouse is ama...|[{document, 0, 46...|[{document, 0, 69...|[{token, 0, 3, Th...|[{word_embeddings...|[{named_entity, 0...|[{chunk, 37, 46, ...|\n|   4.0|we recently had a...|[{document, 0, 79...|[{document, 0, 94...|[{token, 0, 1, we...|[{word_embeddings...|[{named_entity, 0...|[{chunk, 471, 472...|\n|   3.0|Works good for a ...|[{document, 0, 23...|[{document, 0, 12...|[{token, 0, 4, Wo...|[{word_embeddings...|[{named_entity, 0...|[{chunk, 166, 169...|\n+------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+\nonly showing top 4 rows\n\n"]}],"execution_count":0},{"cell_type":"markdown","source":["Spark NLP also offers pretrained pipeline. They are a pre-built set of transformations that have been trained on a large dataset and saved, so they can be reused. This is a great way to get started quickly with NLP tasks like Named Entity Recognition (NER), sentiment analysis, and more.\n\nTo use a pretrained pipeline, you don't need to assemble each individual component. Instead, you can load the entire pipeline at once. One catch is that pretrained pipelines assume that the input column is named \"text\".\n\nHere's an example of how you can do this:"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{},"nuid":"c4715d87-8407-4c63-abc6-5c892fd2b9c5","inputWidgets":{},"title":""}}},{"cell_type":"code","source":["# from sparknlp.pretrained import PretrainedPipeline\n\n# Load a pretrained pipeline\n# pipeline = PretrainedPipeline(\"recognize_entities_dl\", lang=\"en\")\n\n# Rename the target column - \"review\" -> \"text\"\n# text_df_2 = test_df.withColumnRenamed(\"review\", \"text\")\n\n# Use the pipeline to annotate a DataFrame\n# result_df = pipeline.transform(text_df_2)"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{"rowLimit":10000,"byteLimit":2048000},"nuid":"b9e5e56d-2d12-4313-b16c-b69053d49fea","inputWidgets":{},"title":""}},"outputs":[],"execution_count":0},{"cell_type":"markdown","source":["- We import the PretrainedPipeline class from the sparknlp.pretrained module.\n- We create an instance of the PretrainedPipeline class by calling the constructor and passing the name of the pretrained pipeline we want to use (\"recognize_entities_dl\") and the language (\"en\").\n- We rename the input column from \"review\" to \"text\".\n- We use the annotate method of the PretrainedPipeline instance to transform our DataFrame. The annotate method takes two arguments: the DataFrame to transform, and the name of the column in the DataFrame that contains the text to process."],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{},"nuid":"36bb1be1-3124-4a19-a633-2949f3516d8e","inputWidgets":{},"title":""}}},{"cell_type":"code","source":["named_entities = ner_result_df.select(\"entities.result\").withColumnRenamed(\"result\", \"named_entities\")\n\nnamed_entities.show(5, truncate=False)"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{"rowLimit":10000,"byteLimit":2048000},"nuid":"c7d122c4-c8d1-4d15-90a5-a4a9e2d4ab62","inputWidgets":{},"title":""}},"outputs":[{"output_type":"stream","output_type":"stream","name":"stdout","text":["+------------------------+\n|named_entities          |\n+------------------------+\n|[]                      |\n|[Razer Naga, R.A.T, IRL]|\n|[TV, Difference]        |\n|[Said, Catan]           |\n|[zipper]                |\n+------------------------+\nonly showing top 5 rows\n\n"]}],"execution_count":0},{"cell_type":"markdown","source":["### Sentiment analysis\n\n\nNext, we build another pipeline, this time for sentiment analysis. This pipeline is similar to our NER pipeline, but instead of the NER model, we use a pretrained sentiment analysis model."],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{},"nuid":"8405d927-3675-41e0-9589-75805d14e03f","inputWidgets":{},"title":""}}},{"cell_type":"code","source":["document_assembler = DocumentAssembler() \\\n    .setInputCol(\"review\") \\\n    .setOutputCol(\"document\")\n\nsentence_detector = SentenceDetector() \\\n    .setInputCols([\"document\"]) \\\n    .setOutputCol(\"sentences\")\n\ntokenizer = Tokenizer() \\\n    .setInputCols([\"sentences\"]) \\\n    .setOutputCol(\"tokens\")\n\nembeddings = UniversalSentenceEncoder.pretrained('tfhub_use', lang='en') \\\n    .setInputCols([\"document\"]) \\\n    .setOutputCol(\"sentence_embeddings\")\n\nsentiment_model = SentimentDLModel.pretrained(\"sentimentdl_use_imdb\", \"en\") \\\n    .setInputCols([\"sentence_embeddings\"]) \\\n    .setOutputCol(\"sentiment\")\n\npipeline = Pipeline(stages=[\n    document_assembler,\n    sentence_detector,\n    tokenizer,\n    embeddings,\n    sentiment_model\n])"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{"rowLimit":10000,"byteLimit":2048000},"nuid":"3aa705f7-543b-42d9-b018-ce1f0151a742","inputWidgets":{},"title":""}},"outputs":[{"output_type":"stream","output_type":"stream","name":"stdout","text":["tfhub_use download started this may take some time.\nApproximate size to download 923.7 MB\n\r[ | ]\r[OK!]\nsentimentdl_use_imdb download started this may take some time.\nApproximate size to download 12 MB\n\r[ | ]\r[OK!]\n"]}],"execution_count":0},{"cell_type":"markdown","source":["This pipeline first assembles the text into a document, then breaks the document into sentences and tokenizes those sentences.  \nThe pre-trained BertSentenceEmbeddings model then creates embeddings for each sentence, and finally, the SentimentDLModel uses these embeddings to predict sentiment.\n\nEXERCISE: Why did we use sentence embeddings instead of word embeddings like we had done in the NER task?"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{},"nuid":"d1af1439-c852-4f5b-aeec-fd342b69c7e3","inputWidgets":{},"title":""}}},{"cell_type":"markdown","source":["We can now fit and transform our filtered reviews DataFrame using this new pipeline."],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{"rowLimit":10000,"byteLimit":2048000},"nuid":"0c798b10-dd19-46fe-8b7a-ad5e8001901c","inputWidgets":{},"title":""}}},{"cell_type":"code","source":["pipeline_model = pipeline.fit(text_df)\nsentiment_result_df = pipeline_model.transform(text_df)"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{"rowLimit":10000,"byteLimit":2048000},"nuid":"19d52005-b1d4-4dbb-b5fe-b204419289ac","inputWidgets":{},"title":""}},"outputs":[],"execution_count":0},{"cell_type":"code","source":["sentiment_result_df.show(5)"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{"rowLimit":10000,"byteLimit":2048000},"nuid":"82a32ce7-b55e-46ea-85ca-1a686c09bbf5","inputWidgets":{},"title":""}},"outputs":[{"output_type":"stream","output_type":"stream","name":"stdout","text":["+------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+\n|rating|              review|            document|           sentences|              tokens| sentence_embeddings|           sentiment|\n+------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+\n|   4.0|Worked as expecte...|[{document, 0, 10...|[{document, 0, 62...|[{token, 0, 5, Wo...|[{sentence_embedd...|[{category, 0, 10...|\n|   5.0|This mouse is ama...|[{document, 0, 46...|[{document, 0, 69...|[{token, 0, 3, Th...|[{sentence_embedd...|[{category, 0, 46...|\n|   4.0|we recently had a...|[{document, 0, 79...|[{document, 0, 94...|[{token, 0, 1, we...|[{sentence_embedd...|[{category, 0, 79...|\n|   3.0|Works good for a ...|[{document, 0, 23...|[{document, 0, 12...|[{token, 0, 4, Wo...|[{sentence_embedd...|[{category, 0, 23...|\n|   2.0|Fabric is nice an...|[{document, 0, 14...|[{document, 0, 66...|[{token, 0, 5, Fa...|[{sentence_embedd...|[{category, 0, 14...|\n+------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+\nonly showing top 5 rows\n\n"]}],"execution_count":0},{"cell_type":"code","source":["sentiment_result_df.select(\"review\", \"sentiment.result\").show(5, truncate=False)"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{"rowLimit":10000,"byteLimit":2048000},"nuid":"bb514281-dbe6-41af-8a62-1d19ece6d407","inputWidgets":{},"title":""}},"outputs":[{"output_type":"stream","output_type":"stream","name":"stdout","text":["+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------+\n|review                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |result|\n+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------+\n|Worked as expected.I'm not sure what else you expect me to say.  I expected no less.Dunno what else to say.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |[neg] |\n|This mouse is amazing, I had owned a Razer Naga, and a R.A.T. 7 mouse. And I am not afraid to say that this mouse takes the cake, for $20 it looks like its worth $80. The only issue i had was that the seller sent me a wireless version, but I don't mind because it works just as well. Also where you see silver on the mouse in reality it is glossy black, and the mouse looks better IRL then in the pictures. I will add on to this review in a month or so to check in.                                                                                                                                                                                                                                                                                                                                         |[pos] |\n|we recently had a baby boy so now the use of my home theater system is on the shelf for awhile.  My wife also is not into big sound so decided to try out these wireless headphones.  at first i almost sent them back because i took the easy route and hooked them into my receivers headphone jack.  The sound was terrible and there was the dreaded hissing noise you get with some wireless electronics.  before packing them up i decided to hook them up into the jacks on the TV. Difference was like night and day.  the sound quality is great and really brings out some of the background sounds and music that you typically don't notice unless you are in a movie theater.  They aren't super comfortable for long durations but that can be expected with most headphones.  these come fully recommended.|[pos] |\n|Works good for a boy of 6 and his parents.This game is not really requiring any skills (be it game skills, memory or else). It is still fun but gets repetitive fast. Said that, my son still enjoyed it enough to want a proper \"adult\" Catan.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |[pos] |\n|Fabric is nice and soft but zipper broke the first time we used it. Very disappointing. The fit was fine, so we still use it to get our monies worth                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |[pos] |\n+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------+\nonly showing top 5 rows\n\n"]}],"execution_count":0}],"metadata":{"application/vnd.databricks.v1+notebook":{"notebookName":"Module 3: Advanced NLP with Spark NLP","dashboards":[],"notebookMetadata":{"pythonIndentUnit":4},"language":"python","widgets":{}}},"nbformat":4,"nbformat_minor":0}