Module 2 PySpark for NLP.ipynb

{"cells":[{"cell_type":"markdown","source":["In this module, we'll discuss using PySpark for NLP tasks. We'll cover how to load, preprocess, and analyze text data using PySpark. We'll also discuss when to use PySpark for NLP tasks and when to consider other Python NLP libraries.\n\nWe'll introduce Spark NLP, a popular open-source NLP library built on top of PySpark by John Snow Labs. The hands-on exercise will demonstrate how to perform text preprocessing and feature extraction with Spark NLP."],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{},"nuid":"6dd8ea60-b7bb-43b3-b7e5-2726f0605992","inputWidgets":{},"title":""}}},{"cell_type":"code","source":["# !pip install spark-nlp"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{"rowLimit":10000,"byteLimit":2048000},"nuid":"4971880e-2fef-454a-83d0-b079239f5b8c","inputWidgets":{},"title":""}},"outputs":[],"execution_count":0},{"cell_type":"markdown","source":["Now, let's perform the necessary imports:"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{},"nuid":"c2925c1f-3490-45fc-b71c-6be0ed1e959a","inputWidgets":{},"title":""}}},{"cell_type":"code","source":["from pyspark.ml.feature import StopWordsRemover, CountVectorizer, IDF\nfrom pyspark.sql.functions import col, udf\nfrom pyspark.sql.types import StringType\n\n# import sparknlp\n# spark = sparknlp.start()"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{"rowLimit":10000,"byteLimit":2048000},"nuid":"129a0db5-9244-44f1-8e46-c9be56ad0571","inputWidgets":{},"title":""}},"outputs":[],"execution_count":0},{"cell_type":"code","source":["%fs\n\nhead databricks-datasets/amazon/README.md"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{"rowLimit":10000,"byteLimit":2048000},"nuid":"9406bbb4-99df-4183-afb6-6b40fca25175","inputWidgets":{},"title":""}},"outputs":[{"output_type":"display_data","metadata":{"application/vnd.databricks.v1+output":{"datasetInfos":[],"data":"<div class=\"ansiout\"># Amazon Reviews datasets\nThe `data20K` and `test4K` datasets were created by [Professor Julian McAuley](http://cseweb.ucsd.edu/~jmcauley/) at the University of California San Diego with the permission for use in the `databricks-datasets` bucket by Databricks users.\n\nSource: *Image-based recommendations on styles and substitutes.*\nJ. McAuley, C. Targett, J. Shi, A. van den Hengel.\nSIGIR, 2015.\n</div>","removedWidgets":[],"addedWidgets":{},"metadata":{"isDbfsCommandResult":false},"type":"html","arguments":{}}},"output_type":"display_data","data":{"text/html":["<style scoped>\n","  .ansiout {\n","    display: block;\n","    unicode-bidi: embed;\n","    white-space: pre-wrap;\n","    word-wrap: break-word;\n","    word-break: break-all;\n","    font-family: \"Source Code Pro\", \"Menlo\", monospace;;\n","    font-size: 13px;\n","    color: #555;\n","    margin-left: 4px;\n","    line-height: 19px;\n","  }\n","</style>\n","<div class=\"ansiout\"># Amazon Reviews datasets\n","The `data20K` and `test4K` datasets were created by [Professor Julian McAuley](http://cseweb.ucsd.edu/~jmcauley/) at the University of California San Diego with the permission for use in the `databricks-datasets` bucket by Databricks users.\n","\n","Source: *Image-based recommendations on styles and substitutes.*\n","J. McAuley, C. Targett, J. Shi, A. van den Hengel.\n","SIGIR, 2015.\n","</div>"]}}],"execution_count":0},{"cell_type":"code","source":["# dbutils.fs.ls('/amazon')"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{"rowLimit":10000,"byteLimit":2048000},"nuid":"9b9989ac-2975-41d3-b965-9e8df4d9c6ad","inputWidgets":{},"title":""}},"outputs":[],"execution_count":0},{"cell_type":"markdown","source":["**Load a text dataset as a DataFrame**\n\nLet's load a text dataset from the Databricks file system as a DataFrame:"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{},"nuid":"f40ec2c6-64e2-49d1-b4a8-89351e04d44a","inputWidgets":{},"title":""}}},{"cell_type":"code","source":["text_data_path = \"dbfs:/databricks-datasets/amazon/data20K/\"\ntext_df = spark.read.parquet(text_data_path, header=True, inferSchema=True)"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{"rowLimit":10000,"byteLimit":2048000},"nuid":"15d5b2fd-d345-4da4-91fa-a04f08f398ed","inputWidgets":{},"title":""}},"outputs":[],"execution_count":0},{"cell_type":"code","source":["text_df.show(5, truncate=False)"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{"rowLimit":10000,"byteLimit":2048000},"nuid":"21b2dddf-0d70-4a48-a932-1e09d5c8b0f1","inputWidgets":{},"title":""}},"outputs":[{"output_type":"stream","output_type":"stream","name":"stdout","text":["+------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n|rating|review                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |\n+------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n|4.0   |Worked as expected.I'm not sure what else you expect me to say.  I expected no less.Dunno what else to say.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |\n|5.0   |This mouse is amazing, I had owned a Razer Naga, and a R.A.T. 7 mouse. And I am not afraid to say that this mouse takes the cake, for $20 it looks like its worth $80. The only issue i had was that the seller sent me a wireless version, but I don't mind because it works just as well. Also where you see silver on the mouse in reality it is glossy black, and the mouse looks better IRL then in the pictures. I will add on to this review in a month or so to check in.                                                                                                                                                                                                                                                                                                                                         |\n|4.0   |we recently had a baby boy so now the use of my home theater system is on the shelf for awhile.  My wife also is not into big sound so decided to try out these wireless headphones.  at first i almost sent them back because i took the easy route and hooked them into my receivers headphone jack.  The sound was terrible and there was the dreaded hissing noise you get with some wireless electronics.  before packing them up i decided to hook them up into the jacks on the TV. Difference was like night and day.  the sound quality is great and really brings out some of the background sounds and music that you typically don't notice unless you are in a movie theater.  They aren't super comfortable for long durations but that can be expected with most headphones.  these come fully recommended.|\n|3.0   |Works good for a boy of 6 and his parents.This game is not really requiring any skills (be it game skills, memory or else). It is still fun but gets repetitive fast. Said that, my son still enjoyed it enough to want a proper \"adult\" Catan.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |\n|2.0   |Fabric is nice and soft but zipper broke the first time we used it. Very disappointing. The fit was fine, so we still use it to get our monies worth                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |\n+------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\nonly showing top 5 rows\n\n"]}],"execution_count":0},{"cell_type":"markdown","source":["**Preprocess and analyze text data using PySpark**\n\nNow, we'll preprocess and analyze the text data using PySpark. First, tokenize the text.\n\nTokenization is the process of breaking text into individual words or tokens. It's one of the essential steps in NLP to convert unstructured text data into a structured format."],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{},"nuid":"9ab3a580-0940-443b-8838-b46b66dbed29","inputWidgets":{},"title":""}}},{"cell_type":"code","source":["from pyspark.ml.feature import Tokenizer\n\ntokenizer = Tokenizer(inputCol=\"review\", outputCol=\"tokens\")\ntokenized_df = tokenizer.transform(text_df)"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{"rowLimit":10000,"byteLimit":2048000},"nuid":"d5084a61-256a-4e3c-8a91-9ecedebea207","inputWidgets":{},"title":""}},"outputs":[],"execution_count":0},{"cell_type":"code","source":["tokenized_df.show(5, truncate=False)"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{"rowLimit":10000,"byteLimit":2048000},"nuid":"820425b8-16e5-4569-ac14-79dc108cf20c","inputWidgets":{},"title":""}},"outputs":[{"output_type":"stream","output_type":"stream","name":"stdout","text":["+------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n|rating|review                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |tokens                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |\n+------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n|4.0   |Worked as expected.I'm not sure what else you expect me to say.  I expected no less.Dunno what else to say.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |[worked, as, expected.i'm, not, sure, what, else, you, expect, me, to, say., , i, expected, no, less.dunno, what, else, to, say.]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 |\n|5.0   |This mouse is amazing, I had owned a Razer Naga, and a R.A.T. 7 mouse. And I am not afraid to say that this mouse takes the cake, for $20 it looks like its worth $80. The only issue i had was that the seller sent me a wireless version, but I don't mind because it works just as well. Also where you see silver on the mouse in reality it is glossy black, and the mouse looks better IRL then in the pictures. I will add on to this review in a month or so to check in.                                                                                                                                                                                                                                                                                                                                         |[this, mouse, is, amazing,, i, had, owned, a, razer, naga,, and, a, r.a.t., 7, mouse., and, i, am, not, afraid, to, say, that, this, mouse, takes, the, cake,, for, $20, it, looks, like, its, worth, $80., the, only, issue, i, had, was, that, the, seller, sent, me, a, wireless, version,, but, i, don't, mind, because, it, works, just, as, well., also, where, you, see, silver, on, the, mouse, in, reality, it, is, glossy, black,, and, the, mouse, looks, better, irl, then, in, the, pictures., i, will, add, on, to, this, review, in, a, month, or, so, to, check, in.]                                                                                                                                                                                                                                                                                                                                                                                             |\n|4.0   |we recently had a baby boy so now the use of my home theater system is on the shelf for awhile.  My wife also is not into big sound so decided to try out these wireless headphones.  at first i almost sent them back because i took the easy route and hooked them into my receivers headphone jack.  The sound was terrible and there was the dreaded hissing noise you get with some wireless electronics.  before packing them up i decided to hook them up into the jacks on the TV. Difference was like night and day.  the sound quality is great and really brings out some of the background sounds and music that you typically don't notice unless you are in a movie theater.  They aren't super comfortable for long durations but that can be expected with most headphones.  these come fully recommended.|[we, recently, had, a, baby, boy, so, now, the, use, of, my, home, theater, system, is, on, the, shelf, for, awhile., , my, wife, also, is, not, into, big, sound, so, decided, to, try, out, these, wireless, headphones., , at, first, i, almost, sent, them, back, because, i, took, the, easy, route, and, hooked, them, into, my, receivers, headphone, jack., , the, sound, was, terrible, and, there, was, the, dreaded, hissing, noise, you, get, with, some, wireless, electronics., , before, packing, them, up, i, decided, to, hook, them, up, into, the, jacks, on, the, tv., difference, was, like, night, and, day., , the, sound, quality, is, great, and, really, brings, out, some, of, the, background, sounds, and, music, that, you, typically, don't, notice, unless, you, are, in, a, movie, theater., , they, aren't, super, comfortable, for, long, durations, but, that, can, be, expected, with, most, headphones., , these, come, fully, recommended.]|\n|3.0   |Works good for a boy of 6 and his parents.This game is not really requiring any skills (be it game skills, memory or else). It is still fun but gets repetitive fast. Said that, my son still enjoyed it enough to want a proper \"adult\" Catan.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |[works, good, for, a, boy, of, 6, and, his, parents.this, game, is, not, really, requiring, any, skills, (be, it, game, skills,, memory, or, else)., it, is, still, fun, but, gets, repetitive, fast., said, that,, my, son, still, enjoyed, it, enough, to, want, a, proper, \"adult\", catan.]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |\n|2.0   |Fabric is nice and soft but zipper broke the first time we used it. Very disappointing. The fit was fine, so we still use it to get our monies worth                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |[fabric, is, nice, and, soft, but, zipper, broke, the, first, time, we, used, it., very, disappointing., the, fit, was, fine,, so, we, still, use, it, to, get, our, monies, worth]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |\n+------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\nonly showing top 5 rows\n\n"]}],"execution_count":0},{"cell_type":"markdown","source":["##### Remove\n\nStop words are common words that don't carry much meaning and are often removed from text data to reduce noise and computational complexity. Examples of stop words are \"a\", \"an\", \"the\", \"and\", etc."],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{},"nuid":"4986d4fd-f3c5-4811-b73a-a4e5824f03a2","inputWidgets":{},"title":""}}},{"cell_type":"code","source":["remover = StopWordsRemover(inputCol=\"tokens\", outputCol=\"filtered_tokens\")\nfiltered_df = remover.transform(tokenized_df)"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{"rowLimit":10000,"byteLimit":2048000},"nuid":"fe41fb69-751b-4111-b896-ae9ee8c77b1b","inputWidgets":{},"title":""}},"outputs":[],"execution_count":0},{"cell_type":"code","source":["filtered_df.show(5)"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{"rowLimit":10000,"byteLimit":2048000},"nuid":"5f7a62d0-181f-4237-8453-6ebac6d6d62b","inputWidgets":{},"title":""}},"outputs":[{"output_type":"stream","output_type":"stream","name":"stdout","text":["+------+--------------------+--------------------+--------------------+\n|rating|              review|              tokens|     filtered_tokens|\n+------+--------------------+--------------------+--------------------+\n|   4.0|Worked as expecte...|[worked, as, expe...|[worked, expected...|\n|   5.0|This mouse is ama...|[this, mouse, is,...|[mouse, amazing,,...|\n|   4.0|we recently had a...|[we, recently, ha...|[recently, baby, ...|\n|   3.0|Works good for a ...|[works, good, for...|[works, good, boy...|\n|   2.0|Fabric is nice an...|[fabric, is, nice...|[fabric, nice, so...|\n+------+--------------------+--------------------+--------------------+\nonly showing top 5 rows\n\n"]}],"execution_count":0},{"cell_type":"code","source":["cv = CountVectorizer(inputCol=\"filtered_tokens\", outputCol=\"raw_features\")\ncv_model = cv.fit(filtered_df)\nfeaturized_df = cv_model.transform(filtered_df)\n\nidf = IDF(inputCol=\"raw_features\", outputCol=\"features\")\nidf_model = idf.fit(featurized_df)\nresult_df = idf_model.transform(featurized_df)"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{"rowLimit":10000,"byteLimit":2048000},"nuid":"3b040855-66c7-4af3-b80b-4747ae1bb079","inputWidgets":{},"title":""}},"outputs":[],"execution_count":0},{"cell_type":"code","source":["result_df.show(5)"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{"rowLimit":10000,"byteLimit":2048000},"nuid":"b8d82ac0-11a2-41c8-b996-2ca074dbdf1f","inputWidgets":{},"title":""}},"outputs":[{"output_type":"stream","output_type":"stream","name":"stdout","text":["+------+--------------------+--------------------+--------------------+--------------------+--------------------+\n|rating|              review|              tokens|     filtered_tokens|        raw_features|            features|\n+------+--------------------+--------------------+--------------------+--------------------+--------------------+\n|   4.0|Worked as expecte...|[worked, as, expe...|[worked, expected...|(84704,[0,67,98,3...|(84704,[0,67,98,3...|\n|   5.0|This mouse is ama...|[this, mouse, is,...|[mouse, amazing,,...|(84704,[2,17,19,3...|(84704,[2,17,19,3...|\n|   4.0|we recently had a...|[we, recently, ha...|[recently, baby, ...|(84704,[0,2,3,5,6...|(84704,[0,2,3,5,6...|\n|   3.0|Works good for a ...|[works, good, for...|[works, good, boy...|(84704,[4,8,17,27...|(84704,[4,8,17,27...|\n|   2.0|Fabric is nice an...|[fabric, is, nice...|[fabric, nice, so...|(84704,[5,6,7,16,...|(84704,[5,6,7,16,...|\n+------+--------------------+--------------------+--------------------+--------------------+--------------------+\nonly showing top 5 rows\n\n"]}],"execution_count":0},{"cell_type":"markdown","source":["**Build a Spark NLP pipeline**\n\nA pipeline is a sequence of NLP operations applied to text data. In Spark NLP, you create a pipeline by chaining together various annotators and transformers."],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{},"nuid":"9b656f6a-d1df-4353-9be5-b4df68852c37","inputWidgets":{},"title":""}}},{"cell_type":"code","source":["import sparknlp\nfrom sparknlp.annotator import *\nfrom sparknlp.base import *"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{"rowLimit":10000,"byteLimit":2048000},"nuid":"5891a18b-4807-4010-97bf-6ea3fad03b62","inputWidgets":{},"title":""}},"outputs":[],"execution_count":0},{"cell_type":"code","source":["# document_assembler_demo = DocumentAssembler().setInputCol(\"review\").setOutputCol(\"document\")\n# document_assembler_demo_df = document_assembler_demo.transform(text_df)\n\n# document_assembler_demo_df.show(1, truncate=False)\n"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{"rowLimit":10000,"byteLimit":2048000},"nuid":"fe8b1a56-52be-47b4-9f9d-93d4564e55bc","inputWidgets":{},"title":""}},"outputs":[],"execution_count":0},{"cell_type":"code","source":["from pyspark.ml import Pipeline\n\ndocument_assembler = DocumentAssembler().setInputCol(\"review\").setOutputCol(\"document\")\n\ntokenizer = Tokenizer() \\\n    .setInputCols([\"document\"]) \\\n    .setOutputCol(\"tokens\")\n\nnormalizer = Normalizer() \\\n    .setInputCols([\"tokens\"]) \\\n    .setOutputCol(\"normalized\")\n\nlemmatizer = LemmatizerModel.pretrained() \\\n    .setInputCols([\"normalized\"]) \\\n    .setOutputCol(\"lemmas\")\n\nfinisher = Finisher() \\\n    .setInputCols([\"lemmas\"]) \\\n    .setCleanAnnotations(False)\n\npipeline = Pipeline(stages=[\n    document_assembler,\n    tokenizer,\n    normalizer,\n    lemmatizer,\n    finisher\n])"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{"rowLimit":10000,"byteLimit":2048000},"nuid":"95da2587-f293-4d88-ae6c-0ab6dce3d9fa","inputWidgets":{},"title":""}},"outputs":[{"output_type":"stream","output_type":"stream","name":"stdout","text":["lemma_antbnc download started this may take some time.\nApproximate size to download 907.6 KB\n\r[ | ]\r[OK!]\n"]}],"execution_count":0},{"cell_type":"markdown","source":["- DocumentAssembler: It is the first stage in the Spark NLP pipeline, converting input text data into Spark NLP \"Document\" format.\n- Tokenizer: It takes the \"Document\" as input and tokenizes it, breaking the text into individual words or tokens.\n- Normalizer: It removes punctuations, numbers, and any other non-alphabetic characters from the tokens, resulting in clean tokens.\n- Lemmatizer: It reduces words to their base or dictionary form, also known as lemmas. This step helps in standardizing words with similar meanings to their base form, which can improve text analysis.\n- Finisher: It is the final stage in the Spark NLP pipeline, converting the output of the previous annotators and transformers into a DataFrame format that can be used for further analysis or machine learning tasks.\n\n\nTransform the text DataFrame using the Spark NLP pipeline:"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{},"nuid":"335b06d5-6656-4f65-a26a-704612f28723","inputWidgets":{},"title":""}}},{"cell_type":"code","source":["pipeline_model = pipeline.fit(text_df)\nprocessed_df = pipeline_model.transform(text_df)"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{"rowLimit":10000,"byteLimit":2048000},"nuid":"609cbe77-b367-4f01-adc2-b7bbc28b4cb9","inputWidgets":{},"title":""}},"outputs":[],"execution_count":0},{"cell_type":"code","source":["processed_df.show()"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{"rowLimit":10000,"byteLimit":2048000},"nuid":"e790f94b-1089-4d3e-86db-31a8ec6663be","inputWidgets":{},"title":""}},"outputs":[{"output_type":"stream","output_type":"stream","name":"stdout","text":["+------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+\n|rating|              review|            document|              tokens|          normalized|              lemmas|     finished_lemmas|\n+------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+\n|   4.0|Worked as expecte...|[{document, 0, 10...|[{token, 0, 5, Wo...|[{token, 0, 5, Wo...|[{token, 0, 5, Wo...|[Worked, as, expe...|\n|   5.0|This mouse is ama...|[{document, 0, 46...|[{token, 0, 3, Th...|[{token, 0, 3, Th...|[{token, 0, 3, Th...|[This, mouse, be,...|\n|   4.0|we recently had a...|[{document, 0, 79...|[{token, 0, 1, we...|[{token, 0, 1, we...|[{token, 0, 1, we...|[we, recently, ha...|\n|   3.0|Works good for a ...|[{document, 0, 23...|[{token, 0, 4, Wo...|[{token, 0, 4, Wo...|[{token, 0, 4, Wo...|[Works, good, for...|\n|   2.0|Fabric is nice an...|[{document, 0, 14...|[{token, 0, 5, Fa...|[{token, 0, 5, Fa...|[{token, 0, 5, Fa...|[Fabric, be, nice...|\n|   4.0|They are screws w...|[{document, 0, 11...|[{token, 0, 3, Th...|[{token, 0, 3, Th...|[{token, 0, 3, Th...|[They, be, screw,...|\n|   5.0|I love this prold...|[{document, 0, 14...|[{token, 0, 0, I,...|[{token, 0, 0, I,...|[{token, 0, 0, I,...|[I, love, this, p...|\n|   3.0|This is an ok pum...|[{document, 0, 86...|[{token, 0, 3, Th...|[{token, 0, 3, Th...|[{token, 0, 3, Th...|[This, be, an, ok...|\n|   5.0|Perfect for the g...|[{document, 0, 11...|[{token, 0, 6, Pe...|[{token, 0, 6, Pe...|[{token, 0, 6, Pe...|[Perfect, for, th...|\n|   4.0|I'd hoped my 2 ye...|[{document, 0, 19...|[{token, 0, 2, I'...|[{token, 0, 1, Id...|[{token, 0, 1, Id...|[Id, hope, i, yea...|\n|   5.0|This had some ass...|[{document, 0, 31...|[{token, 0, 3, Th...|[{token, 0, 3, Th...|[{token, 0, 3, Th...|[This, have, some...|\n|   5.0|Worked like a cha...|[{document, 0, 76...|[{token, 0, 5, Wo...|[{token, 0, 5, Wo...|[{token, 0, 5, Wo...|[Worked, like, a,...|\n|   5.0|Sounds quality is...|[{document, 0, 23...|[{token, 0, 5, So...|[{token, 0, 5, So...|[{token, 0, 5, So...|[Sounds, quality,...|\n|   5.0|Great fun!  Very ...|[{document, 0, 11...|[{token, 0, 4, Gr...|[{token, 0, 4, Gr...|[{token, 0, 4, Gr...|[Great, fun, Very...|\n|   5.0|Hubby loves this ...|[{document, 0, 11...|[{token, 0, 4, Hu...|[{token, 0, 4, Hu...|[{token, 0, 4, Hu...|[Hubby, love, thi...|\n|   5.0|It fits the 15-85...|[{document, 0, 35...|[{token, 0, 1, It...|[{token, 0, 1, It...|[{token, 0, 1, It...|[It, fit, the, le...|\n|   1.0|In theory this pr...|[{document, 0, 17...|[{token, 0, 1, In...|[{token, 0, 1, In...|[{token, 0, 1, In...|[In, theory, this...|\n|   3.0|The bottom of the...|[{document, 0, 50...|[{token, 0, 2, Th...|[{token, 0, 2, Th...|[{token, 0, 2, Th...|[The, bottom, of,...|\n|   5.0|Love kingston mem...|[{document, 0, 29...|[{token, 0, 3, Lo...|[{token, 0, 3, Lo...|[{token, 0, 3, Lo...|[Love, kingston, ...|\n|   4.0|These are an inno...|[{document, 0, 14...|[{token, 0, 4, Th...|[{token, 0, 4, Th...|[{token, 0, 4, Th...|[These, be, an, i...|\n+------+--------------------+--------------------+--------------------+--------------------+--------------------+--------------------+\nonly showing top 20 rows\n\n"]}],"execution_count":0},{"cell_type":"code","source":["processed_df.select(\"review\", \"finished_lemmas\").show(4, truncate=False)"],"metadata":{"application/vnd.databricks.v1+cell":{"showTitle":false,"cellMetadata":{"rowLimit":10000,"byteLimit":2048000},"nuid":"8b50b304-e7eb-4b21-9ca4-094403385345","inputWidgets":{},"title":""}},"outputs":[],"execution_count":0}],"metadata":{"application/vnd.databricks.v1+notebook":{"notebookName":"Module 2: PySpark for NLP","dashboards":[],"notebookMetadata":{"pythonIndentUnit":4,"mostRecentlyExecutedCommandWithImplicitDF":{"commandId":948763216766329,"dataframes":["_sqldf"]}},"language":"python","widgets":{}}},"nbformat":4,"nbformat_minor":0}