polmineR-examples.ipynb

{"cells":[{"metadata":{"trusted":false},"cell_type":"markdown","source":"## polmineR\n\nIn this Jupyter notebook we look at various examples to work with the [polmineR](https://github.com/PolMine/polmineR) package. These examples are taken from the [README documentation](https://github.com/PolMine/polmineR#core-functions) about the core functions present in [polmineR](https://github.com/PolMine/polmineR).\n\nTo run individual cells in this notebook, you can go to the cell and press SHIFT+ENTER. You can make changes as you like and run the cells again interactively.\n\nIf you would like to run the whole notebook, you can click on the Kernel tab in the options menu and select \"Restart and Run all\". This will restart the R kernel and run all the cells in the notebook.\n\nYou can learn more about Jupyter notebooks at https://jupyter.org."},{"metadata":{"trusted":true},"cell_type":"code","source":"library(polmineR)\nuse(\"polmineR\")\nuse(\"europarl\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"id":"fabulous-mayor","cell_type":"code","source":"corpus()","execution_count":null,"outputs":[]},{"metadata":{},"id":"adaptive-plaza","cell_type":"markdown","source":"### partition (and partition_bundle)\n\nAll methods can be applied to a whole corpus, as well as to partitions (i.e. subcorpora). Use the metadata of a corpus (so-called s-attributes) to define a subcorpus."},{"metadata":{"trusted":true},"id":"restricted-paint","cell_type":"code","source":"ep2005 <- partition(\"EUROPARL-EN\", text_year = \"2006\")\nsize(ep2005)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"id":"governing-certification","cell_type":"code","source":"barroso <- partition(\"EUROPARL-EN\", speaker_name = \"Barroso\", regex = TRUE)\nsize(barroso)","execution_count":null,"outputs":[]},{"metadata":{},"id":"forty-steps","cell_type":"markdown","source":"Partitions can be bundled into partition_bundle objects, and most methods can be applied to a whole corpus, a partition, or a partition_bundle object alike. Consult the package vignette to learn more."},{"metadata":{},"id":"running-playing","cell_type":"markdown","source":"### count (using CQP syntax)\n\nCounting occurrences of a feature in a corpus, a partition or in the partitions of a partition_bundle is a basic operation. By offering access to the query syntax of the Corpus Query Processor (CQP), polmineR package exposes a query syntax that goes far beyond regular expressions. See the [CQP documentation](http://www.ims.uni-stuttgart.de/forschung/projekte/CorpusWorkbench/CQPTutorial/cqp-tutorial.2up.pdf) to learn more."},{"metadata":{"trusted":true},"id":"optical-graphic","cell_type":"code","source":"count(\"EUROPARL-EN\", \"France\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"id":"virtual-rebel","cell_type":"code","source":"count(\"EUROPARL-EN\", c(\"France\", \"Germany\", \"Britain\", \"Spain\", \"Italy\", \"Denmark\", \"Poland\"))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"id":"capable-camcorder","cell_type":"code","source":"count(\"EUROPARL-EN\", '\"[pP]opulism\"')","execution_count":null,"outputs":[]},{"metadata":{},"id":"colored-donor","cell_type":"markdown","source":"### dispersion (across one or two dimensions)\n\nThe dispersion method is there to analyse the dispersion of a query, or a set of queries across one or two dimensions (absolute and relative frequencies). The CQP syntax can be used."},{"metadata":{"trusted":true},"id":"enclosed-chosen","cell_type":"code","source":"populism <- dispersion(\"EUROPARL-EN\", \"populism\", s_attribute = \"text_year\", progress = FALSE)\npop_regex <- dispersion(\"EUROPARL-EN\", '\"[pP]opulism\"', s_attribute = \"text_year\", cqp = TRUE, progress = FALSE)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"id":"damaged-uniform","cell_type":"code","source":"populism","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"id":"interpreted-wednesday","cell_type":"code","source":"pop_regex","execution_count":null,"outputs":[]},{"metadata":{},"id":"norwegian-phone","cell_type":"markdown","source":"### cooccurrences (to analyse collocations)\n\nThe cooccurrences method is used to analyse the context of a query (including some statistics)."},{"metadata":{"trusted":true},"id":"spare-cotton","cell_type":"code","source":"islam <- cooccurrences(\"EUROPARL-EN\", query = 'Islam', left = 10, right = 10)\nislam <- subset(islam, rank_ll <= 100)\ndotplot(islam)\n","execution_count":null,"outputs":[]},{"metadata":{},"id":"natural-basis","cell_type":"markdown","source":"### features (keyword extraction)\n\nCompare partitions to identify features / keywords (using statistical tests such as chi square)."},{"metadata":{"trusted":true},"id":"guilty-halifax","cell_type":"code","source":"ep_2002 <- partition(\"EUROPARL-EN\", text_year = \"2002\", p_attribute = \"word\")\nep_pre_2002 <- partition(\"EUROPARL-EN\", text_year = 1997:2001, p_attribute = \"word\")\nfeatures(ep_2002, ep_pre_2002, included = FALSE) %>%\n  subset(rank_chisquare <= 10) %>%\n  format() %>%\n  knitr::kable(format = \"markdown\")\n","execution_count":null,"outputs":[]},{"metadata":{},"id":"female-printing","cell_type":"markdown","source":"### kwic (also known as concordances)\n\nSo what happens in the context of a word, or a CQP query? To attain valid research results, reading will often be necessary."},{"metadata":{"trusted":true},"id":"smoking-applicant","cell_type":"code","source":"kwic(\"EUROPARL-EN\", \"Islam\", meta = c(\"text_date\", \"speaker_name\")) %>%\n  as.data.frame() %>%\n  .[1:8,] %>%\n  knitr::kable(format = \"markdown\", escape = FALSE)\n","execution_count":null,"outputs":[]},{"metadata":{},"id":"valued-mission","cell_type":"markdown","source":"### as.TermDocumentMatrix (for text mining purposes)\n\nMany advanced methods in text mining require term document matrices as input. Based on the metadata of a corpus, these data structures can be obtained in a fast and flexible manner, for performing topic modelling, machine learning etc."},{"metadata":{"trusted":true},"id":"renewable-concrete","cell_type":"code","source":"speakers <- partition_bundle(\n  \"EUROPARL-EN\", s_attribute = \"speaker_id\",\n  progress = FALSE, verbose = FALSE\n)\nspeakers_count <- count(speakers, p_attribute = \"word\", progress = TRUE)\ntdm <- as.TermDocumentMatrix(speakers_count, col = \"count\")\ndim(tdm)\n","execution_count":null,"outputs":[]},{"metadata":{},"id":"seven-clinic","cell_type":"markdown","source":"### read (the full text)\n\nCorpus analysis involves moving from text to numbers, and back again. Use the read method, to inspect the full text of a partition (a speech given by chancellor Angela Merkel in this case)."},{"metadata":{"trusted":true},"id":"mexican-skiing","cell_type":"code","source":"library(GermaParl)\nuse(\"GermaParl\")\nif (!\"GERMAPARL\" %in% corpus()$corpus){\n  GermaParl::germaparl_download_corpus()\n  use(\"GermaParl\")\n}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"id":"extreme-collaboration","cell_type":"code","source":"merkel <- partition(\"GERMAPARL\", speaker = \"Angela Merkel\", date = \"2013-09-03\")\nread(merkel)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"id":"helpful-surfing","cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"name":"ir","display_name":"R","language":"R"},"language_info":{"name":"R","codemirror_mode":"r","pygments_lexer":"r","mimetype":"text/x-r-source","file_extension":".r","version":"4.1.0"}},"nbformat":4,"nbformat_minor":5}