3-1-Bigrams-Stemming-and-Lemmatizing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Bigrams, Stemming and Lemmatizing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Exploring the `reuters` corpus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'       The Reuters-21578 benchmark corpus, ApteMod version  This is a publically available version of the well-known Reuters-21578 \"ApteMod\" corpus for text categorization.  It has been used in publications like these:   * Yiming Yang and X. Liu. \"A re-examination of text categorization    methods\".  1999.  Proceedings of 22nd Annual International SIGIR.    http://citeseer.nj.nec.com/yang99reexamination.html   * Thorsten Joachims. \"Text categorization with support vector    machines: learning with many relevant features\".  1998. Proceedings    of ECML-98, 10th European Conference on Machine Learning.    http://citeseer.nj.nec.com/joachims98text.html  ApteMod is a collection of 10,788 documents from the Reuters financial newswire service, partitioned into a training set with 7769 documents and a test set with 3019 documents.  The total size of the corpus is about 43 MB.  It is also available for download from http://kdd.ics.uci.edu/databases/reuters21578/reuters21578.html , which includes a more extensive history of the data revisions.  The distribution of categories in the ApteMod corpus is highly skewed, with 36.7% of the documents in the most common category, and only 0.0185% (2 documents) in each of the five least common categories. In fact, the original data source is even more skewed---in creating the corpus, any categories that did not contain at least one document in the training set and one document in the test set were removed from the corpus by its original creator.  In the ApteMod corpus, each document belongs to one or more categories.  There are 90 categories in the corpus.  The average number of categories per document is 1.235, and the average number of documents per category is about 148, or 1.37% of the corpus.   -Ken Williams   ken@mathforum.org           Copyright & Notification   (extracted from the README at the UCI address above)  The copyright for the text of newswire articles and Reuters annotations in the Reuters-21578 collection resides with Reuters Ltd. Reuters Ltd. and Carnegie Group, Inc. have agreed to allow the free distribution of this data *for research purposes only*.    If you publish results based on this data set, please acknowledge its use, refer to the data set by the name \"Reuters-21578, Distribution 1.0\", and inform your readers of the current location of the data set (see \"Availability & Questions\").  '"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from nltk.corpus import reuters\n",
    "\n",
    "reuters.readme().replace('\\n', ' ')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['test/14826',\n",
       " 'test/14828',\n",
       " 'test/14829',\n",
       " 'test/14832',\n",
       " 'test/14833',\n",
       " 'test/14839',\n",
       " 'test/14840',\n",
       " 'test/14841',\n",
       " 'test/14842',\n",
       " 'test/14843',\n",
       " 'test/14844',\n",
       " 'test/14849',\n",
       " 'test/14852',\n",
       " 'test/14854',\n",
       " 'test/14858',\n",
       " 'test/14859',\n",
       " 'test/14860',\n",
       " 'test/14861',\n",
       " 'test/14862',\n",
       " 'test/14863',\n",
       " 'test/14865',\n",
       " 'test/14867',\n",
       " 'test/14872',\n",
       " 'test/14873',\n",
       " 'test/14875',\n",
       " 'test/14876',\n",
       " 'test/14877',\n",
       " 'test/14881',\n",
       " 'test/14882',\n",
       " 'test/14885',\n",
       " 'test/14886',\n",
       " 'test/14888',\n",
       " 'test/14890',\n",
       " 'test/14891',\n",
       " 'test/14892',\n",
       " 'test/14899',\n",
       " 'test/14900',\n",
       " 'test/14903',\n",
       " 'test/14904',\n",
       " 'test/14907',\n",
       " 'test/14909',\n",
       " 'test/14911',\n",
       " 'test/14912',\n",
       " 'test/14913',\n",
       " 'test/14918',\n",
       " 'test/14919',\n",
       " 'test/14921',\n",
       " 'test/14922',\n",
       " 'test/14923',\n",
       " 'test/14926',\n",
       " 'test/14928',\n",
       " 'test/14930',\n",
       " 'test/14931',\n",
       " 'test/14932',\n",
       " 'test/14933',\n",
       " 'test/14934',\n",
       " 'test/14941',\n",
       " 'test/14943',\n",
       " 'test/14949',\n",
       " 'test/14951',\n",
       " 'test/14954',\n",
       " 'test/14957',\n",
       " 'test/14958',\n",
       " 'test/14959',\n",
       " 'test/14960',\n",
       " 'test/14962',\n",
       " 'test/14963',\n",
       " 'test/14964',\n",
       " 'test/14965',\n",
       " 'test/14967',\n",
       " 'test/14968',\n",
       " 'test/14969',\n",
       " 'test/14970',\n",
       " 'test/14971',\n",
       " 'test/14974',\n",
       " 'test/14975',\n",
       " 'test/14978',\n",
       " 'test/14981',\n",
       " 'test/14982',\n",
       " 'test/14983',\n",
       " 'test/14984',\n",
       " 'test/14985',\n",
       " 'test/14986',\n",
       " 'test/14987',\n",
       " 'test/14988',\n",
       " 'test/14993',\n",
       " 'test/14995',\n",
       " 'test/14998',\n",
       " 'test/15000',\n",
       " 'test/15001',\n",
       " 'test/15002',\n",
       " 'test/15004',\n",
       " 'test/15005',\n",
       " 'test/15006',\n",
       " 'test/15011',\n",
       " 'test/15012',\n",
       " 'test/15013',\n",
       " 'test/15016',\n",
       " 'test/15017',\n",
       " 'test/15020',\n",
       " 'test/15023',\n",
       " 'test/15024',\n",
       " 'test/15026',\n",
       " 'test/15027',\n",
       " 'test/15028',\n",
       " 'test/15029',\n",
       " 'test/15031',\n",
       " 'test/15032',\n",
       " 'test/15033',\n",
       " 'test/15037',\n",
       " 'test/15038',\n",
       " 'test/15043',\n",
       " 'test/15045',\n",
       " 'test/15046',\n",
       " 'test/15048',\n",
       " 'test/15049',\n",
       " 'test/15052',\n",
       " 'test/15053',\n",
       " 'test/15055',\n",
       " 'test/15056',\n",
       " 'test/15060',\n",
       " 'test/15061',\n",
       " 'test/15062',\n",
       " 'test/15063',\n",
       " 'test/15065',\n",
       " 'test/15067',\n",
       " 'test/15069',\n",
       " 'test/15070',\n",
       " 'test/15074',\n",
       " 'test/15077',\n",
       " 'test/15078',\n",
       " 'test/15079',\n",
       " 'test/15082',\n",
       " 'test/15090',\n",
       " 'test/15091',\n",
       " 'test/15092',\n",
       " 'test/15093',\n",
       " 'test/15094',\n",
       " 'test/15095',\n",
       " 'test/15096',\n",
       " 'test/15097',\n",
       " 'test/15103',\n",
       " 'test/15104',\n",
       " 'test/15106',\n",
       " 'test/15107',\n",
       " 'test/15109',\n",
       " 'test/15110',\n",
       " 'test/15111',\n",
       " 'test/15112',\n",
       " 'test/15118',\n",
       " 'test/15119',\n",
       " 'test/15120',\n",
       " 'test/15121',\n",
       " 'test/15122',\n",
       " 'test/15124',\n",
       " 'test/15126',\n",
       " 'test/15128',\n",
       " 'test/15129',\n",
       " 'test/15130',\n",
       " 'test/15132',\n",
       " 'test/15136',\n",
       " 'test/15138',\n",
       " 'test/15141',\n",
       " 'test/15144',\n",
       " 'test/15145',\n",
       " 'test/15146',\n",
       " 'test/15149',\n",
       " 'test/15152',\n",
       " 'test/15153',\n",
       " 'test/15154',\n",
       " 'test/15156',\n",
       " 'test/15157',\n",
       " 'test/15161',\n",
       " 'test/15162',\n",
       " 'test/15171',\n",
       " 'test/15172',\n",
       " 'test/15175',\n",
       " 'test/15179',\n",
       " 'test/15180',\n",
       " 'test/15185',\n",
       " 'test/15188',\n",
       " 'test/15189',\n",
       " 'test/15190',\n",
       " 'test/15193',\n",
       " 'test/15194',\n",
       " 'test/15197',\n",
       " 'test/15198',\n",
       " 'test/15200',\n",
       " 'test/15204',\n",
       " 'test/15205',\n",
       " 'test/15206',\n",
       " 'test/15207',\n",
       " 'test/15208',\n",
       " 'test/15210',\n",
       " 'test/15211',\n",
       " 'test/15212',\n",
       " 'test/15213',\n",
       " 'test/15217',\n",
       " 'test/15219',\n",
       " 'test/15220',\n",
       " 'test/15221',\n",
       " 'test/15222',\n",
       " 'test/15223',\n",
       " 'test/15226',\n",
       " 'test/15227',\n",
       " 'test/15230',\n",
       " 'test/15233',\n",
       " 'test/15234',\n",
       " 'test/15237',\n",
       " 'test/15238',\n",
       " 'test/15239',\n",
       " 'test/15240',\n",
       " 'test/15242',\n",
       " 'test/15243',\n",
       " 'test/15244',\n",
       " 'test/15246',\n",
       " 'test/15247',\n",
       " 'test/15250',\n",
       " 'test/15253',\n",
       " 'test/15254',\n",
       " 'test/15255',\n",
       " 'test/15258',\n",
       " 'test/15259',\n",
       " 'test/15262',\n",
       " 'test/15263',\n",
       " 'test/15264',\n",
       " 'test/15265',\n",
       " 'test/15270',\n",
       " 'test/15271',\n",
       " 'test/15273',\n",
       " 'test/15274',\n",
       " 'test/15276',\n",
       " 'test/15278',\n",
       " 'test/15280',\n",
       " 'test/15281',\n",
       " 'test/15283',\n",
       " 'test/15287',\n",
       " 'test/15290',\n",
       " 'test/15292',\n",
       " 'test/15294',\n",
       " 'test/15295',\n",
       " 'test/15296',\n",
       " 'test/15299',\n",
       " 'test/15300',\n",
       " 'test/15302',\n",
       " 'test/15303',\n",
       " 'test/15306',\n",
       " 'test/15307',\n",
       " 'test/15308',\n",
       " 'test/15309',\n",
       " 'test/15310',\n",
       " 'test/15311',\n",
       " 'test/15312',\n",
       " 'test/15313',\n",
       " 'test/15314',\n",
       " 'test/15315',\n",
       " 'test/15321',\n",
       " 'test/15322',\n",
       " 'test/15324',\n",
       " 'test/15325',\n",
       " 'test/15326',\n",
       " 'test/15327',\n",
       " 'test/15329',\n",
       " 'test/15335',\n",
       " 'test/15336',\n",
       " 'test/15337',\n",
       " 'test/15339',\n",
       " 'test/15341',\n",
       " 'test/15344',\n",
       " 'test/15345',\n",
       " 'test/15348',\n",
       " 'test/15349',\n",
       " 'test/15351',\n",
       " 'test/15352',\n",
       " 'test/15354',\n",
       " 'test/15356',\n",
       " 'test/15357',\n",
       " 'test/15359',\n",
       " 'test/15363',\n",
       " 'test/15364',\n",
       " 'test/15365',\n",
       " 'test/15366',\n",
       " 'test/15367',\n",
       " 'test/15368',\n",
       " 'test/15372',\n",
       " 'test/15375',\n",
       " 'test/15378',\n",
       " 'test/15379',\n",
       " 'test/15380',\n",
       " 'test/15383',\n",
       " 'test/15384',\n",
       " 'test/15386',\n",
       " 'test/15387',\n",
       " 'test/15388',\n",
       " 'test/15389',\n",
       " 'test/15391',\n",
       " 'test/15394',\n",
       " 'test/15396',\n",
       " 'test/15397',\n",
       " 'test/15400',\n",
       " 'test/15404',\n",
       " 'test/15406',\n",
       " 'test/15409',\n",
       " 'test/15410',\n",
       " 'test/15411',\n",
       " 'test/15413',\n",
       " 'test/15415',\n",
       " 'test/15416',\n",
       " 'test/15417',\n",
       " 'test/15420',\n",
       " 'test/15421',\n",
       " 'test/15424',\n",
       " 'test/15425',\n",
       " 'test/15427',\n",
       " 'test/15428',\n",
       " 'test/15429',\n",
       " 'test/15430',\n",
       " 'test/15431',\n",
       " 'test/15432',\n",
       " 'test/15436',\n",
       " 'test/15438',\n",
       " 'test/15441',\n",
       " 'test/15442',\n",
       " 'test/15444',\n",
       " 'test/15446',\n",
       " 'test/15447',\n",
       " 'test/15448',\n",
       " 'test/15449',\n",
       " 'test/15450',\n",
       " 'test/15451',\n",
       " 'test/15452',\n",
       " 'test/15453',\n",
       " 'test/15454',\n",
       " 'test/15455',\n",
       " 'test/15457',\n",
       " 'test/15459',\n",
       " 'test/15460',\n",
       " 'test/15462',\n",
       " 'test/15464',\n",
       " 'test/15467',\n",
       " 'test/15468',\n",
       " 'test/15471',\n",
       " 'test/15472',\n",
       " 'test/15476',\n",
       " 'test/15477',\n",
       " 'test/15478',\n",
       " 'test/15479',\n",
       " 'test/15481',\n",
       " 'test/15482',\n",
       " 'test/15483',\n",
       " 'test/15484',\n",
       " 'test/15485',\n",
       " 'test/15487',\n",
       " 'test/15489',\n",
       " 'test/15494',\n",
       " 'test/15495',\n",
       " 'test/15496',\n",
       " 'test/15500',\n",
       " 'test/15501',\n",
       " 'test/15503',\n",
       " 'test/15504',\n",
       " 'test/15510',\n",
       " 'test/15511',\n",
       " 'test/15515',\n",
       " 'test/15520',\n",
       " 'test/15521',\n",
       " 'test/15522',\n",
       " 'test/15523',\n",
       " 'test/15527',\n",
       " 'test/15528',\n",
       " 'test/15531',\n",
       " 'test/15532',\n",
       " 'test/15535',\n",
       " 'test/15536',\n",
       " 'test/15539',\n",
       " 'test/15540',\n",
       " 'test/15542',\n",
       " 'test/15543',\n",
       " 'test/15544',\n",
       " 'test/15545',\n",
       " 'test/15547',\n",
       " 'test/15548',\n",
       " 'test/15549',\n",
       " 'test/15550',\n",
       " 'test/15551',\n",
       " 'test/15552',\n",
       " 'test/15553',\n",
       " 'test/15556',\n",
       " 'test/15558',\n",
       " 'test/15559',\n",
       " 'test/15560',\n",
       " 'test/15561',\n",
       " 'test/15562',\n",
       " 'test/15563',\n",
       " 'test/15565',\n",
       " 'test/15566',\n",
       " 'test/15567',\n",
       " 'test/15568',\n",
       " 'test/15569',\n",
       " 'test/15570',\n",
       " 'test/15571',\n",
       " 'test/15572',\n",
       " 'test/15573',\n",
       " 'test/15574',\n",
       " 'test/15575',\n",
       " 'test/15578',\n",
       " 'test/15579',\n",
       " 'test/15580',\n",
       " 'test/15581',\n",
       " 'test/15582',\n",
       " 'test/15583',\n",
       " 'test/15584',\n",
       " 'test/15585',\n",
       " 'test/15590',\n",
       " 'test/15591',\n",
       " 'test/15593',\n",
       " 'test/15594',\n",
       " 'test/15595',\n",
       " 'test/15596',\n",
       " 'test/15597',\n",
       " 'test/15598',\n",
       " 'test/15600',\n",
       " 'test/15601',\n",
       " 'test/15602',\n",
       " 'test/15603',\n",
       " 'test/15605',\n",
       " 'test/15607',\n",
       " 'test/15610',\n",
       " 'test/15613',\n",
       " 'test/15615',\n",
       " 'test/15616',\n",
       " 'test/15617',\n",
       " 'test/15618',\n",
       " 'test/15620',\n",
       " 'test/15621',\n",
       " 'test/15623',\n",
       " 'test/15624',\n",
       " 'test/15625',\n",
       " 'test/15626',\n",
       " 'test/15629',\n",
       " 'test/15632',\n",
       " 'test/15634',\n",
       " 'test/15636',\n",
       " 'test/15637',\n",
       " 'test/15639',\n",
       " 'test/15640',\n",
       " 'test/15641',\n",
       " 'test/15642',\n",
       " 'test/15643',\n",
       " 'test/15646',\n",
       " 'test/15648',\n",
       " 'test/15649',\n",
       " 'test/15651',\n",
       " 'test/15653',\n",
       " 'test/15655',\n",
       " 'test/15656',\n",
       " 'test/15664',\n",
       " 'test/15666',\n",
       " 'test/15667',\n",
       " 'test/15668',\n",
       " 'test/15669',\n",
       " 'test/15672',\n",
       " 'test/15674',\n",
       " 'test/15675',\n",
       " 'test/15676',\n",
       " 'test/15677',\n",
       " 'test/15679',\n",
       " 'test/15680',\n",
       " 'test/15682',\n",
       " 'test/15686',\n",
       " 'test/15688',\n",
       " 'test/15689',\n",
       " 'test/15691',\n",
       " 'test/15692',\n",
       " 'test/15694',\n",
       " 'test/15695',\n",
       " 'test/15696',\n",
       " 'test/15698',\n",
       " 'test/15702',\n",
       " 'test/15703',\n",
       " 'test/15704',\n",
       " 'test/15707',\n",
       " 'test/15708',\n",
       " 'test/15709',\n",
       " 'test/15710',\n",
       " 'test/15713',\n",
       " 'test/15715',\n",
       " 'test/15717',\n",
       " 'test/15719',\n",
       " 'test/15720',\n",
       " 'test/15721',\n",
       " 'test/15723',\n",
       " 'test/15725',\n",
       " 'test/15726',\n",
       " 'test/15727',\n",
       " 'test/15728',\n",
       " 'test/15729',\n",
       " 'test/15732',\n",
       " 'test/15733',\n",
       " 'test/15736',\n",
       " 'test/15737',\n",
       " 'test/15739',\n",
       " 'test/15742',\n",
       " 'test/15749',\n",
       " 'test/15751',\n",
       " 'test/15753',\n",
       " 'test/15757',\n",
       " 'test/15759',\n",
       " 'test/15762',\n",
       " 'test/15767',\n",
       " 'test/15768',\n",
       " 'test/15769',\n",
       " 'test/15772',\n",
       " 'test/15777',\n",
       " 'test/15778',\n",
       " 'test/15780',\n",
       " 'test/15782',\n",
       " 'test/15785',\n",
       " 'test/15790',\n",
       " 'test/15793',\n",
       " 'test/15797',\n",
       " 'test/15798',\n",
       " 'test/15800',\n",
       " 'test/15801',\n",
       " 'test/15803',\n",
       " 'test/15804',\n",
       " 'test/15805',\n",
       " 'test/15807',\n",
       " 'test/15808',\n",
       " 'test/15810',\n",
       " 'test/15811',\n",
       " 'test/15816',\n",
       " 'test/15817',\n",
       " 'test/15819',\n",
       " 'test/15821',\n",
       " 'test/15822',\n",
       " 'test/15823',\n",
       " 'test/15829',\n",
       " 'test/15831',\n",
       " 'test/15832',\n",
       " 'test/15833',\n",
       " 'test/15834',\n",
       " 'test/15836',\n",
       " 'test/15838',\n",
       " 'test/15840',\n",
       " 'test/15841',\n",
       " 'test/15842',\n",
       " 'test/15844',\n",
       " 'test/15845',\n",
       " 'test/15846',\n",
       " 'test/15847',\n",
       " 'test/15851',\n",
       " 'test/15852',\n",
       " 'test/15853',\n",
       " 'test/15854',\n",
       " 'test/15855',\n",
       " 'test/15856',\n",
       " 'test/15858',\n",
       " 'test/15859',\n",
       " 'test/15860',\n",
       " 'test/15861',\n",
       " 'test/15863',\n",
       " 'test/15864',\n",
       " 'test/15865',\n",
       " 'test/15866',\n",
       " 'test/15867',\n",
       " 'test/15868',\n",
       " 'test/15869',\n",
       " 'test/15870',\n",
       " 'test/15871',\n",
       " 'test/15872',\n",
       " 'test/15874',\n",
       " 'test/15875',\n",
       " 'test/15876',\n",
       " 'test/15877',\n",
       " 'test/15878',\n",
       " 'test/15879',\n",
       " 'test/15881',\n",
       " 'test/15885',\n",
       " 'test/15886',\n",
       " 'test/15888',\n",
       " 'test/15889',\n",
       " 'test/15890',\n",
       " 'test/15892',\n",
       " 'test/15893',\n",
       " 'test/15894',\n",
       " 'test/15895',\n",
       " 'test/15896',\n",
       " 'test/15897',\n",
       " 'test/15898',\n",
       " 'test/15899',\n",
       " 'test/15900',\n",
       " 'test/15901',\n",
       " 'test/15902',\n",
       " 'test/15903',\n",
       " 'test/15904',\n",
       " 'test/15906',\n",
       " 'test/15908',\n",
       " 'test/15909',\n",
       " 'test/15910',\n",
       " 'test/15911',\n",
       " 'test/15912',\n",
       " 'test/15913',\n",
       " 'test/15914',\n",
       " 'test/15916',\n",
       " 'test/15917',\n",
       " 'test/15918',\n",
       " 'test/15920',\n",
       " 'test/15921',\n",
       " 'test/15922',\n",
       " 'test/15923',\n",
       " 'test/15924',\n",
       " 'test/15925',\n",
       " 'test/15927',\n",
       " 'test/15928',\n",
       " 'test/15929',\n",
       " 'test/15930',\n",
       " 'test/15932',\n",
       " 'test/15933',\n",
       " 'test/15934',\n",
       " 'test/15937',\n",
       " 'test/15939',\n",
       " 'test/15942',\n",
       " 'test/15944',\n",
       " 'test/15949',\n",
       " 'test/15950',\n",
       " 'test/15951',\n",
       " 'test/15952',\n",
       " 'test/15953',\n",
       " 'test/15956',\n",
       " 'test/15959',\n",
       " 'test/15960',\n",
       " 'test/15961',\n",
       " 'test/15963',\n",
       " 'test/15964',\n",
       " 'test/15967',\n",
       " 'test/15968',\n",
       " 'test/15969',\n",
       " 'test/15970',\n",
       " 'test/15973',\n",
       " 'test/15975',\n",
       " 'test/15976',\n",
       " 'test/15977',\n",
       " 'test/15978',\n",
       " 'test/15979',\n",
       " 'test/15980',\n",
       " 'test/15981',\n",
       " 'test/15984',\n",
       " 'test/15985',\n",
       " 'test/15987',\n",
       " 'test/15988',\n",
       " 'test/15989',\n",
       " 'test/15993',\n",
       " 'test/15995',\n",
       " 'test/15996',\n",
       " 'test/15997',\n",
       " 'test/15999',\n",
       " 'test/16002',\n",
       " 'test/16003',\n",
       " 'test/16004',\n",
       " 'test/16005',\n",
       " 'test/16006',\n",
       " 'test/16007',\n",
       " 'test/16009',\n",
       " 'test/16012',\n",
       " 'test/16013',\n",
       " 'test/16014',\n",
       " 'test/16015',\n",
       " 'test/16016',\n",
       " 'test/16021',\n",
       " 'test/16022',\n",
       " 'test/16023',\n",
       " 'test/16026',\n",
       " 'test/16029',\n",
       " 'test/16030',\n",
       " 'test/16033',\n",
       " 'test/16037',\n",
       " 'test/16040',\n",
       " 'test/16041',\n",
       " 'test/16045',\n",
       " 'test/16052',\n",
       " 'test/16053',\n",
       " 'test/16055',\n",
       " 'test/16063',\n",
       " 'test/16066',\n",
       " 'test/16067',\n",
       " 'test/16068',\n",
       " 'test/16069',\n",
       " 'test/16071',\n",
       " 'test/16072',\n",
       " 'test/16074',\n",
       " 'test/16075',\n",
       " 'test/16076',\n",
       " 'test/16077',\n",
       " 'test/16079',\n",
       " 'test/16080',\n",
       " 'test/16083',\n",
       " 'test/16086',\n",
       " 'test/16088',\n",
       " 'test/16091',\n",
       " 'test/16093',\n",
       " 'test/16094',\n",
       " 'test/16095',\n",
       " 'test/16096',\n",
       " 'test/16097',\n",
       " 'test/16098',\n",
       " 'test/16099',\n",
       " 'test/16100',\n",
       " 'test/16103',\n",
       " 'test/16106',\n",
       " 'test/16107',\n",
       " 'test/16108',\n",
       " 'test/16110',\n",
       " 'test/16111',\n",
       " 'test/16112',\n",
       " 'test/16115',\n",
       " 'test/16117',\n",
       " 'test/16118',\n",
       " 'test/16119',\n",
       " 'test/16120',\n",
       " 'test/16122',\n",
       " 'test/16123',\n",
       " 'test/16125',\n",
       " 'test/16126',\n",
       " 'test/16130',\n",
       " 'test/16133',\n",
       " 'test/16134',\n",
       " 'test/16136',\n",
       " 'test/16139',\n",
       " 'test/16140',\n",
       " 'test/16141',\n",
       " 'test/16142',\n",
       " 'test/16143',\n",
       " 'test/16144',\n",
       " 'test/16145',\n",
       " 'test/16146',\n",
       " 'test/16147',\n",
       " 'test/16148',\n",
       " 'test/16149',\n",
       " 'test/16150',\n",
       " 'test/16152',\n",
       " 'test/16155',\n",
       " 'test/16158',\n",
       " 'test/16159',\n",
       " 'test/16161',\n",
       " 'test/16162',\n",
       " 'test/16163',\n",
       " 'test/16164',\n",
       " 'test/16166',\n",
       " 'test/16170',\n",
       " 'test/16171',\n",
       " 'test/16172',\n",
       " 'test/16173',\n",
       " 'test/16175',\n",
       " 'test/16176',\n",
       " 'test/16177',\n",
       " 'test/16179',\n",
       " 'test/16180',\n",
       " 'test/16185',\n",
       " 'test/16188',\n",
       " 'test/16189',\n",
       " 'test/16190',\n",
       " 'test/16193',\n",
       " 'test/16194',\n",
       " 'test/16195',\n",
       " 'test/16196',\n",
       " 'test/16197',\n",
       " 'test/16200',\n",
       " 'test/16201',\n",
       " 'test/16202',\n",
       " 'test/16203',\n",
       " 'test/16206',\n",
       " 'test/16207',\n",
       " 'test/16210',\n",
       " 'test/16211',\n",
       " 'test/16212',\n",
       " 'test/16213',\n",
       " 'test/16214',\n",
       " 'test/16215',\n",
       " 'test/16216',\n",
       " 'test/16219',\n",
       " 'test/16221',\n",
       " 'test/16223',\n",
       " 'test/16225',\n",
       " 'test/16226',\n",
       " 'test/16228',\n",
       " 'test/16230',\n",
       " 'test/16232',\n",
       " 'test/16233',\n",
       " 'test/16234',\n",
       " 'test/16236',\n",
       " 'test/16238',\n",
       " 'test/16241',\n",
       " 'test/16243',\n",
       " 'test/16244',\n",
       " 'test/16246',\n",
       " 'test/16247',\n",
       " 'test/16248',\n",
       " 'test/16250',\n",
       " 'test/16251',\n",
       " 'test/16252',\n",
       " 'test/16255',\n",
       " 'test/16256',\n",
       " 'test/16257',\n",
       " 'test/16258',\n",
       " 'test/16260',\n",
       " 'test/16262',\n",
       " 'test/16263',\n",
       " 'test/16264',\n",
       " 'test/16265',\n",
       " 'test/16266',\n",
       " 'test/16268',\n",
       " 'test/16269',\n",
       " 'test/16270',\n",
       " 'test/16271',\n",
       " 'test/16274',\n",
       " 'test/16275',\n",
       " 'test/16277',\n",
       " 'test/16278',\n",
       " 'test/16279',\n",
       " 'test/16281',\n",
       " 'test/16282',\n",
       " 'test/16283',\n",
       " 'test/16284',\n",
       " 'test/16285',\n",
       " 'test/16286',\n",
       " 'test/16287',\n",
       " 'test/16288',\n",
       " 'test/16289',\n",
       " 'test/16291',\n",
       " 'test/16294',\n",
       " 'test/16297',\n",
       " 'test/16298',\n",
       " 'test/16299',\n",
       " 'test/16300',\n",
       " 'test/16301',\n",
       " 'test/16302',\n",
       " 'test/16303',\n",
       " 'test/16304',\n",
       " 'test/16307',\n",
       " 'test/16310',\n",
       " 'test/16311',\n",
       " 'test/16312',\n",
       " 'test/16314',\n",
       " 'test/16315',\n",
       " 'test/16316',\n",
       " 'test/16317',\n",
       " 'test/16318',\n",
       " 'test/16319',\n",
       " 'test/16320',\n",
       " 'test/16324',\n",
       " 'test/16327',\n",
       " 'test/16331',\n",
       " 'test/16332',\n",
       " 'test/16336',\n",
       " 'test/16337',\n",
       " 'test/16339',\n",
       " 'test/16342',\n",
       " 'test/16343',\n",
       " 'test/16346',\n",
       " 'test/16347',\n",
       " 'test/16348',\n",
       " 'test/16350',\n",
       " 'test/16354',\n",
       " 'test/16357',\n",
       " 'test/16359',\n",
       " 'test/16360',\n",
       " 'test/16362',\n",
       " 'test/16363',\n",
       " 'test/16365',\n",
       " 'test/16366',\n",
       " 'test/16367',\n",
       " 'test/16369',\n",
       " 'test/16370',\n",
       " 'test/16371',\n",
       " 'test/16372',\n",
       " 'test/16374',\n",
       " 'test/16376',\n",
       " 'test/16377',\n",
       " 'test/16379',\n",
       " 'test/16380',\n",
       " 'test/16383',\n",
       " 'test/16385',\n",
       " 'test/16386',\n",
       " 'test/16388',\n",
       " 'test/16390',\n",
       " 'test/16392',\n",
       " 'test/16393',\n",
       " 'test/16394',\n",
       " 'test/16395',\n",
       " 'test/16396',\n",
       " 'test/16398',\n",
       " 'test/16399',\n",
       " 'test/16400',\n",
       " 'test/16401',\n",
       " 'test/16402',\n",
       " 'test/16403',\n",
       " 'test/16404',\n",
       " 'test/16405',\n",
       " 'test/16406',\n",
       " 'test/16407',\n",
       " 'test/16409',\n",
       " 'test/16410',\n",
       " 'test/16415',\n",
       " 'test/16417',\n",
       " 'test/16418',\n",
       " 'test/16419',\n",
       " 'test/16420',\n",
       " 'test/16421',\n",
       " 'test/16422',\n",
       " 'test/16424',\n",
       " 'test/16426',\n",
       " 'test/16427',\n",
       " 'test/16428',\n",
       " 'test/16429',\n",
       " 'test/16430',\n",
       " 'test/16432',\n",
       " 'test/16433',\n",
       " 'test/16434',\n",
       " 'test/16437',\n",
       " 'test/16438',\n",
       " 'test/16440',\n",
       " 'test/16441',\n",
       " 'test/16442',\n",
       " 'test/16443',\n",
       " 'test/16444',\n",
       " 'test/16448',\n",
       " 'test/16449',\n",
       " 'test/16450',\n",
       " 'test/16454',\n",
       " 'test/16457',\n",
       " 'test/16458',\n",
       " 'test/16459',\n",
       " 'test/16460',\n",
       " 'test/16461',\n",
       " 'test/16463',\n",
       " 'test/16465',\n",
       " 'test/16468',\n",
       " 'test/16469',\n",
       " 'test/16470',\n",
       " 'test/16471',\n",
       " 'test/16472',\n",
       " 'test/16473',\n",
       " 'test/16475',\n",
       " 'test/16476',\n",
       " 'test/16478',\n",
       " 'test/16479',\n",
       " 'test/16480',\n",
       " 'test/16481',\n",
       " 'test/16483',\n",
       " 'test/16486',\n",
       " 'test/16487',\n",
       " 'test/16488',\n",
       " 'test/16490',\n",
       " 'test/16492',\n",
       " 'test/16493',\n",
       " 'test/16495',\n",
       " 'test/16496',\n",
       " 'test/16499',\n",
       " 'test/16502',\n",
       " 'test/16505',\n",
       " 'test/16510',\n",
       " 'test/16512',\n",
       " 'test/16513',\n",
       " 'test/16518',\n",
       " 'test/16519',\n",
       " 'test/16521',\n",
       " 'test/16522',\n",
       " 'test/16523',\n",
       " 'test/16525',\n",
       " 'test/16527',\n",
       " 'test/16530',\n",
       " 'test/16531',\n",
       " 'test/16533',\n",
       " 'test/16538',\n",
       " 'test/16539',\n",
       " 'test/16545',\n",
       " 'test/16546',\n",
       " 'test/16549',\n",
       " 'test/16551',\n",
       " 'test/16554',\n",
       " 'test/16555',\n",
       " 'test/16561',\n",
       " 'test/16563',\n",
       " 'test/16564',\n",
       " 'test/16565',\n",
       " 'test/16568',\n",
       " 'test/16569',\n",
       " 'test/16570',\n",
       " 'test/16574',\n",
       " 'test/16577',\n",
       " 'test/16581',\n",
       " 'test/16583',\n",
       " 'test/16584',\n",
       " 'test/16585',\n",
       " 'test/16587',\n",
       " 'test/16588',\n",
       " 'test/16589',\n",
       " 'test/16590',\n",
       " 'test/16591',\n",
       " ...]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reuters.fileids()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'training/9995'"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reuters.fileids()[-1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10788"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(reuters.fileids())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['acq',\n",
       " 'alum',\n",
       " 'barley',\n",
       " 'bop',\n",
       " 'carcass',\n",
       " 'castor-oil',\n",
       " 'cocoa',\n",
       " 'coconut',\n",
       " 'coconut-oil',\n",
       " 'coffee',\n",
       " 'copper',\n",
       " 'copra-cake',\n",
       " 'corn',\n",
       " 'cotton',\n",
       " 'cotton-oil',\n",
       " 'cpi',\n",
       " 'cpu',\n",
       " 'crude',\n",
       " 'dfl',\n",
       " 'dlr',\n",
       " 'dmk',\n",
       " 'earn',\n",
       " 'fuel',\n",
       " 'gas',\n",
       " 'gnp',\n",
       " 'gold',\n",
       " 'grain',\n",
       " 'groundnut',\n",
       " 'groundnut-oil',\n",
       " 'heat',\n",
       " 'hog',\n",
       " 'housing',\n",
       " 'income',\n",
       " 'instal-debt',\n",
       " 'interest',\n",
       " 'ipi',\n",
       " 'iron-steel',\n",
       " 'jet',\n",
       " 'jobs',\n",
       " 'l-cattle',\n",
       " 'lead',\n",
       " 'lei',\n",
       " 'lin-oil',\n",
       " 'livestock',\n",
       " 'lumber',\n",
       " 'meal-feed',\n",
       " 'money-fx',\n",
       " 'money-supply',\n",
       " 'naphtha',\n",
       " 'nat-gas',\n",
       " 'nickel',\n",
       " 'nkr',\n",
       " 'nzdlr',\n",
       " 'oat',\n",
       " 'oilseed',\n",
       " 'orange',\n",
       " 'palladium',\n",
       " 'palm-oil',\n",
       " 'palmkernel',\n",
       " 'pet-chem',\n",
       " 'platinum',\n",
       " 'potato',\n",
       " 'propane',\n",
       " 'rand',\n",
       " 'rape-oil',\n",
       " 'rapeseed',\n",
       " 'reserves',\n",
       " 'retail',\n",
       " 'rice',\n",
       " 'rubber',\n",
       " 'rye',\n",
       " 'ship',\n",
       " 'silver',\n",
       " 'sorghum',\n",
       " 'soy-meal',\n",
       " 'soy-oil',\n",
       " 'soybean',\n",
       " 'strategic-metal',\n",
       " 'sugar',\n",
       " 'sun-meal',\n",
       " 'sun-oil',\n",
       " 'sunseed',\n",
       " 'tea',\n",
       " 'tin',\n",
       " 'trade',\n",
       " 'veg-oil',\n",
       " 'wheat',\n",
       " 'wpi',\n",
       " 'yen',\n",
       " 'zinc']"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reuters.categories()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['ASIAN', 'EXPORTERS', 'FEAR', 'DAMAGE', 'FROM', 'U', '.', 'S', '.-', 'JAPAN', 'RIFT', 'Mounting', 'trade', 'friction', 'between', 'the', 'U', '.', 'S', '.', 'And', 'Japan', 'has', 'raised', 'fears', 'among', 'many', 'of', 'Asia', \"'\", 's', 'exporting', 'nations', 'that', 'the', 'row', 'could', 'inflict', 'far', '-', 'reaching', 'economic', 'damage', ',', 'businessmen', 'and', 'officials', 'said', '.'], ['They', 'told', 'Reuter', 'correspondents', 'in', 'Asian', 'capitals', 'a', 'U', '.', 'S', '.', 'Move', 'against', 'Japan', 'might', 'boost', 'protectionist', 'sentiment', 'in', 'the', 'U', '.', 'S', '.', 'And', 'lead', 'to', 'curbs', 'on', 'American', 'imports', 'of', 'their', 'products', '.'], ...]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reuters.sents('test/14826')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Bigrams"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "142723"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trade_words = reuters.words(categories='trade')\n",
    "len(trade_words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['ASIAN', 'EXPORTERS', 'FEAR', 'DAMAGE', 'FROM', 'U', ...]"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trade_words_condensed = trade_words[:100]\n",
    "trade_words_condensed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['asian',\n",
       " 'exporters',\n",
       " 'fear',\n",
       " 'damage',\n",
       " 'u',\n",
       " '.',\n",
       " '.-',\n",
       " 'japan',\n",
       " 'rift',\n",
       " 'mounting']"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from nltk.corpus import stopwords\n",
    "\n",
    "# Remove stopwords from trade_words_condensed and lower case it\n",
    "trade_words_condensed = [w.lower() for w in trade_words_condensed if w.lower() not in stopwords.words('english')]\n",
    "trade_words_condensed[:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['asian',\n",
       " 'exporters',\n",
       " 'fear',\n",
       " 'damage',\n",
       " 'u',\n",
       " 'japan',\n",
       " 'rift',\n",
       " 'mounting',\n",
       " 'trade',\n",
       " 'friction',\n",
       " 'u',\n",
       " 'japan',\n",
       " 'raised',\n",
       " 'fears',\n",
       " 'among',\n",
       " 'many',\n",
       " 'asia',\n",
       " 'exporting',\n",
       " 'nations',\n",
       " 'row',\n",
       " 'could',\n",
       " 'inflict',\n",
       " 'far',\n",
       " 'reaching',\n",
       " 'economic',\n",
       " 'damage',\n",
       " 'businessmen',\n",
       " 'officials',\n",
       " 'said',\n",
       " 'told',\n",
       " 'reuter',\n",
       " 'correspondents',\n",
       " 'asian',\n",
       " 'capitals',\n",
       " 'u',\n",
       " 'move',\n",
       " 'japan',\n",
       " 'might',\n",
       " 'boost',\n",
       " 'protectionist',\n",
       " 'sentiment',\n",
       " 'u',\n",
       " 'lead',\n",
       " 'curbs',\n",
       " 'american',\n",
       " 'imports',\n",
       " 'products',\n",
       " 'exporters',\n",
       " 'said',\n",
       " 'conflict',\n",
       " 'would',\n",
       " 'hurt',\n",
       " 'long']"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import string # Contains string constants eg. ascii_lowercase which is 'a...z', string formatting functions, other string functions like .capwords() and .translate().\n",
    "\n",
    "# Remove punctuation\n",
    "# trade_words_condensed = [w for w in trade_words_condensed if w not in string.punctuation]\n",
    "punct_combo = [c + \"\\\"\" for c in string.punctuation ] + [\"\\\"\" + c for c in string.punctuation] + [\".-\", \":-\", \"..\", \"...\"]\n",
    "trade_words_condensed = [w for w in trade_words_condensed if w not in string.punctuation and w not in punct_combo]\n",
    "trade_words_condensed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('asian', 'exporters'),\n",
       " ('exporters', 'fear'),\n",
       " ('fear', 'damage'),\n",
       " ('damage', 'u'),\n",
       " ('u', 'japan')]"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from nltk import bigrams\n",
    "\n",
    "bi_trade_words_condensed = list(bigrams(trade_words_condensed))\n",
    "bi_trade_words_condensed[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('u', 'japan') 2\n",
      "('asian', 'exporters') 1\n",
      "('exporters', 'fear') 1\n"
     ]
    }
   ],
   "source": [
    "from nltk import FreqDist\n",
    "\n",
    "bi_fdist = FreqDist(bi_trade_words_condensed)\n",
    "\n",
    "for word, frequency in bi_fdist.most_common(3):\n",
    "    print(word, frequency)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAFkCAYAAADPOydvAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvFvnyVgAAIABJREFUeJzt3Xl4FfXZxvHvnYSwLwKiUVAUBVRkC+77UrXWamvVitiqtaUqVrrZ3bbW9u1ia+uKpbbVVlyrda/Vtu47AQQEF1REFGWTNYAsz/vHHNK8vAghyZxJJvfnus6Vc+bMye+Ra8x9ZuY3zygiMDMzAyjJugAzM2s6HApmZlbDoWBmZjUcCmZmVsOhYGZmNRwKZmZWw6FgZmY1HApmZlbDoWBmZjXKsi5gS3Xv3j169+5dr8+uWLGCtm3bNm5BZrV4G7M0NWT7qqqqmh8RW29uvWYXCr1792b8+PH1+mxVVRWVlZWNXJHZf3kbszQ1ZPuS9FZd1vPhIzMzq+FQMDOzGg4FMzOr4VAwM7MaDgUzM6uRWihI6iXpEUnTJb0kafRG1pGkKyTNkDRZ0tC06jEzs81Lc0rqGuAbETFBUkegStLDETGt1jofB3YtPPYBxhR+pmL1Wt9lzsxsU1LbU4iIORExofB8KTAd2H6D1U4A/hKJZ4EukirSqOfBqXM474F5zJi7NI1fb2aWC0W5eE1Sb2AI8NwGb20PvF3r9ezCsjkbfH4kMBKgoqKCqqqqLa7hxmcXsXDlOs78w1P84shutC3z6RRrfNXV1fXaPs3qohjbV+qhIKkDcAfw1YhYsuHbG/nI/zvGExFjgbEAw4YNi/pc0ff7AWs4+tf/YvbStdz2RhmXnzoYaWPDm9Wfr2i2NBVj+0r167KkViSBMC4i7tzIKrOBXrVe9wTeTaOW9q3LuHD/rWhXXso9L77LX5+t0xXfZmYtSpqzjwT8EZgeEZd9xGr3AJ8vzELaF1gcEXM+Yt0G69mpjF98ZiAAl9w3jYmzPkhrKDOzZinNPYUDgM8Bh0uaVHgcK+kcSecU1nkAeAOYAfwBOC/FegA4ftB2nLl/b1avDUaNm8DC5R+mPaSZWbOR2jmFiHiSjZ8zqL1OAKPSquGjfO/Y3Xhx9iImzlrE6Fsmcv1Ze1Na4vMLZmYtcgpOeVkJV582lK7ty3nitflc8e/Xsi7JzKxJaJGhALBdl7aFGUhwxX9e49FX5mZdkplZ5lpsKAActOvWfPWIvkTAV2+dxDuLVmRdkplZplp0KAB85fBdOKTv1iyqXs154yawas3arEsyM8tMiw+FkhLxu88OZvsubXnx7UX87P7pWZdkZpaZFh8KAFu1L+fqEUNpVSr+8sxb3D3pnaxLMjPLhEOhYHCvLvzwuN0B+M4dU3jtfTfOM7OWx6FQy+n77sgJg7djxeq1nHNjFctWrcm6JDOzonIo1CKJn5+4J7v26MDr85bznTsmk1xfZ2bWMjgUNtCuvIwxp1fSvryU+ybP4YanZ2ZdkplZ0TgUNmKXHh341UmDAPjZA9OpesuN88ysZXAofIRPDKzgrAOSxnnn3zSBBctWZV2SmVnqHAqb8N2P78bQHbowZ/FKRt8yibXrfH7BzPLNobAJ5WUlXD0iaZz35Iz5XP6vV7MuycwsVQ6Fzajo3JYrTh1SaJw3g0fcOM/McsyhUAcH7tqdrx/ZF4Cv3TqJ2R9UZ1yRmVk6HAp1NOqwXTisnxvnmVm+ORTqqKRE/LbQOG/y7MVcct+0rEsyM2t0DoUt0KVdOWNOH0p5aQk3PjuLuya6cZ6Z5YtDYQsN7NmFH34yaZz33Tun8Kob55lZjjgU6mHEPjtw4pDt3TjPzHLHoVAPkvjZp/ek3zYdeWPecr79NzfOM7N8cCjUU9vyUsacPpQOrcu4f8oc/vzUzKxLMjNrMIdCA+y8dQd+ddJAAP7ngemMn7kw44rMzBrGodBAx+5ZwdkH7sSadcGomyYw343zzKwZcyg0gu98vD/DdtyK95esYvQtE904z8yaLYdCI2hVWsJVpw2le4dynpqxgN8+7MZ5ZtY8ORQaybad23DFqUMoEVz1yAz+8/L7WZdkZrbFHAqNaP9duvONo/oB8LVbX+TthW6cZ2bNi0OhkZ17SB+O6N+DxSuSxnkrV7txnpk1H6mFgqQ/SZoraepHvN9Z0r2SXpT0kqSz0qqlmEpKxGWnDKbnVm2Z8s5ifuLGeWbWjKS5p3A9cMwm3h8FTIuIQcChwG8kladYT9F0bteKMSMqKS8r4abnZnHnhNlZl2RmViephUJEPA5s6mquADpKEtChsG5umgjt2bMzFx+/BwDf+/sUXn5vScYVmZltXpbnFK4CdgPeBaYAoyNiXYb1NLpT9+rFZ4b2ZOXqdZx74wSWrlyddUlmZptUluHYRwOTgMOBPsDDkp6IiP/3lVrSSGAkQEVFBVVVVfUasLq6ut6fra/P7BiMf72MN+cv54t/eIwL9+tCsnNkeZTFNmYtRzG2ryxD4SzgF5G0F50h6U2gP/D8hitGxFhgLMCwYcOisrKyXgNWVVVR3882xPV9lnP8lU/y3DurmLSiK188aOei12DFkdU2Zi1DMbavLA8fzQKOAJC0DdAPeCPDelKzU/f2XHpy0jjv5/94mRfcOM/Mmqg0p6TeDDwD9JM0W9LZks6RdE5hlUuA/SVNAf4NfDsi5qdVT9aOGVDBlw7aibXrglHjJjBvqRvnmVnTk9rho4gYvpn33wWOSmv8puhbx/Rn0tuLeGHmB1xw80T+evbelJX6+kEzazr8F6mI/ts4rzXPvLGAy9w4z8yaGIdCkW3TqQ1XDk8a513z6Ov8a5ob55lZ0+FQyMB+fbrxzaOTxnlfv20Ssxa4cZ6ZNQ0OhYycc3AfjtytB0tWruG8m6rcOM/MmgSHQkZKSsRvTh7MDl3bMfWdJVx870tZl2Rm5lDIUud2rbhmxFDKy0q4+fm3+VuVG+eZWbYcChkbsH1nLjkhaZz3/b9PYdq7bpxnZtlxKDQBn91rB06u7MmqNes4b1wVS9w4z8wy4lBoIi751AB2q+jEzAXVfPO2F0laQpmZFZdDoYlo06qUMSOG0rF1GQ9Ne58/PJHLNlBm1sQ5FJqQ3t3b8+tTBgHwywdf4bk3FmRckZm1NA6FJuboPbblywfvzNp1wfk3T2Tu0pVZl2RmLYhDoQm68Oh+7L1TV+YtXcVXbprImrW5uiGdmTVhDoUmqKy0hKuGD6F7h9Y89+ZCfv2QG+eZWXE4FJqoHp3acNVpQygtEdc+9joPu3GemRWBQ6EJ23fnbnyrVuO8txYsz7giM8s7h0ITN/LgnTlq921YunIN5944wY3zzCxVDoUmThKXnjyIHbu1Y9qcJfzobjfOM7P0OBSagc5tWzFmRCWty0q4dfzb3PbC21mXZGY55VBoJnbfrhOXfGoAABfdPZWX3l2ccUVmlkcOhWbklGG9+OywXqxas45zb5zA4hVunGdmjcuh0MxcfMIe7F7RiVkLq/nm7W6cZ2aNy6HQzLRpVcq1p1fSsU0ZD097n98/7sZ5ZtZ4HArN0A7d2nHZKYMB+NWDL/OsG+eZWSNxKDRTH9t9G845pA/rAs6/aSJzl7hxnpk1nEOhGfvmUX3Zd+euzF+2ivNvduM8M2s4h0IzVlZawhXDh9CjY2uef3Mhl/7zlaxLMrNmzqHQzPXo2IarThtKaYn4/eNv8M+X3su6JDNrxhwKObD3Tl35zjH9AfjmbS8yc74b55lZ/TgUcuKLB+3EMXtsy9JVazh3nBvnmVn9OBRyQhK/Onkgvbu1Y/qcJfzgrqm+sM3MtlhqoSDpT5LmSpq6iXUOlTRJ0kuSHkurlpaiU5tWjDm9kjatSvhb1WxudeM8M9tCae4pXA8c81FvSuoCXAMcHxF7ACenWEuLsVtFJ376qT0B+OE9LzH1HTfOM7O6Sy0UIuJxYOEmVjkNuDMiZhXWn5tWLS3NSZU9Gb53Lz5cs45zx1WxuNqN88ysbsoyHLsv0ErSo0BH4PKI+MvGVpQ0EhgJUFFRQVVVVb0GrK6urvdnm5vjewbPvVrGGwtXcPYfHuPbB3ShRMq6rNxrSduYFV8xtq8sQ6EMqASOANoCz0h6NiJe3XDFiBgLjAUYNmxYVFZW1mvAqqoq6vvZ5uiGXar5xBVPMH7OKl5Y1oXzDt0l65Jyr6VtY1Zcxdi+spx9NBt4MCKWR8R84HFgUIb15E6vru347WeTxnm//ucrPP36/IwrMrOmLstQuBs4SFKZpHbAPsD0DOvJpSN224bzDk0a511w80Ted+M8M9uENKek3gw8A/STNFvS2ZLOkXQOQERMBx4EJgPPA9dFxEdOX7X6+/rH+rJ/n27MX/Yh5980gdVunGdmH2GLzylI2groFRGTN7VeRAzf3O+KiEuBS7e0Btsy6xvnfeKKJ3hh5gf86sGX+f4nds+6LDNrguq0pyDpUUmdJHUFXgT+LOmydEuzxtS9Q2uuPm0oZSXiD0+8yYNT52Rdkpk1QXU9fNQ5IpYAJwJ/johK4Mj0yrI0DOvdle98PGmcd+Htk3nTjfPMbAN1DYUySRXAKcB9KdZjKTv7wJ04ds9C47wbq1jxoRvnmdl/1TUULgb+CcyIiBck7Qy8ll5ZlhZJ/PIzA9m5e3tefm8p379rihvnmVmNuobCnIgYGBHnAUTEG4DPKTRTHdu04prTh9KmVQl3TniHm5934zwzS9Q1FK6s4zJrJvpv24n/+XTSOO/H97zElNlunGdmm5mSKmk/YH9ga0lfr/VWJ6A0zcIsfScO7cn4tz7gpudmce64Ku77yoF0aVeedVlmlqHN7SmUAx1IwqNjrccS4KR0S7Ni+OFxu7Pn9p2Z/cEKvn7bi6xb5/MLZi3ZJvcUIuIx4DFJ10fEW0WqyYqoTatSrhkxlOOufJL/vDyXMY+9zqjD3DjPrKWq6zmF1pLGSnpI0n/WP1KtzIqmV9d2/K7QOO83D73CUzPcOM+spaprKNwOTAR+AFxY62E5cVj/Hnzl8F1qGue9t9iN88xaorqGwpqIGBMRz0dE1fpHqpVZ0X31yL4cuEt3Fix34zyzlqquoXCvpPMkVUjquv6RamVWdKUl4vJTB7NtpzaMf+sDfvGPl7MuycyKrK6hcAbJ4aKngarCY3xaRVl2unVozdUjksZ5f3zyTR6Y4sZ5Zi1JnUIhInbayGPntIuzbFTuuBXfO3Y3AL71t8m8Pm9ZxhWZWbHU6X4Kkj6/seUR8ZfGLceairMO6E3VrA+4f/Iczr2xirtGHUC78ixv6W1mxVDXw0d71XocBPwYOD6lmqwJqGmct3V7Xn1/Gd//+1Q3zjNrAep6+OgrtR5fAoaQXO1sOdahdRnXnl5J21al/H3iO4x7blbWJZlZyup7j+ZqYNfGLMSapr7bdOTnJyaN835y7zQmz16UcUVmlqa63o7zXkn3FB73A68Ad6dbmjUVnxqyPafvuwMfrl3HuTdO4IPlH2ZdkpmlpK5nDn9d6/ka4K2ImJ1CPdZEXXTc7kyZvZgXZy/ma7dN4k9n7EVJibIuy8waWV3PKTwGvEzSIXUrwF8VW5jWZaVcPWIoXdq14tFX5nH1IzOyLsnMUlDXw0enAM8DJ5Pcp/k5SW6d3cL03CppnCfBZf96lSdfc+M8s7yp64nm7wN7RcQZEfF5YG/govTKsqbq0H49+MrhuxIBF9wykTmLV2Rdkpk1orqGQklEzK31esEWfNZyZvQRu3LQrt1ZuPxDRo2bwIdr3DjPLC/q+of9QUn/lHSmpDOB+4EH0ivLmrKkcd4QKjq3YcKsRfz8H9OzLsnMGskmQ0HSLpIOiIgLgd8DA4FBwDPA2CLUZ01U1/blXD1iKK1KxZ+fmsl9k9/NuiQzawSb21P4HbAUICLujIivR8TXSPYSfpd2cda0Dd1hK75faJz37b9NZsZcN84za+42Fwq9I2LyhgsjYjzQO5WKrFk5Y//efHLQdiz/cC3n3ljF8lVrsi7JzBpgc6HQZhPvtW3MQqx5ksQvTtyTPlu357W5y/je36e4cZ5ZM7a5UHhB0pc2XCjpbJIb7XwkSX+SNFfS1M2st5ektb7uoflqX2ic1668lLsnvcuNz76VdUlmVk+bC4WvAmdJelTSbwqPx4AvAqM389nrgWM2tYKkUuCXwD/rWK81UbvWbpx33zQmve3GeWbN0SZDISLej4j9gYuBmYXHxRGxX0S8t5nPPg4s3Mz4XwHuAOZuZj1rBk4YvD1n7Lcjq9cGo8a5cZ5Zc1TX3kePRMSVhcd/GmNgSdsDnwaubYzfZ03D9z+xO4N7deGdRSv46q2TWLfO5xfMmpMs76/4O+DbEbFW2nS3TUkjgZEAFRUVVFVt8nTGR6qurq73Z63uzhnYigvfF4+9Oo/vjnucU3bvkHVJReNtzNJUjO1Lac4UkdQbuC8iBmzkvTeB9WnQneTGPSMj4q5N/c5hw4bF+PHj61VPVVUVlZWV9fqsbZnHX53HGX9+HoAbztqbg/tunXFFxeFtzNLUkO1LUlVEDNvcepn1L4qInSKid0T0Bv4GnLe5QLDm4+C+WzP6iKRx3uhbJvLuIjfOM2sOUgsFSTeTtMPoJ2m2pLMlnSPpnLTGtKblgsN35eC+W/NB9WrOc+M8s2YhtXMKETF8C9Y9M606LDslJeJ3nx3McVc8waS3F/E/D0znx8fvkXVZZrYJbn9tqeravpxrTq+kVam4/umZ3POiG+eZNWUOBUvd4F5duOi43QH4zh2Tee39pRlXZGYfxaFgRfG5fXfk+EHbUf3hWs4dN8GN88yaKIeCFYUkfn7inuzSowMz5i7jO3e6cZ5ZU+RQsKJZ3zivfXkp9774Ln95xo3zzJoah4IV1S49OvDLkwYC8NP7pzFh1gcZV2RmtTkUrOiOG7gdZ+7fm9Vrg/PHTWChG+eZNRkOBcvE947djaE7dOHdxSsZfctE1rpxnlmT4FCwTJSXlXD1iKF0bV/OE6/N54p/v5Z1SWaGQ8EyVNG5LZefOhgJrvjPazz6im+rYZY1h4Jl6qBdt+ZrR/YlAr566yTeceM8s0w5FCxz5x+2C4f225pFhcZ5q9aszboksxbLoWCZKykRvz1lMNt3acuLby/ip/dNz7oksxbLoWBNwlbty7lmxFDKS0v467Nvcfekd7IuyaxFcihYkzGoVxcu+uT6xnlTeNWN88yKzqFgTcrp++zApwZvx4rVaznnxiqWuXGeWVE5FKxJkcT/nLgnfbfpwBvzlvPtOya7cZ5ZETkUrMlpV17GmNMr6dC6jPsnz+H6p2dmXZJZi+FQsCapz9Yd+FWhcd7P7p9O1VtunGdWDA4Fa7KO3bOCLxywE2vWBeffNIEFy1ZlXZJZ7jkUrEn77rH9qdxxK+YsXsnoWya5cZ5ZyhwK1qS1Ki3h6tOG0q19OU/OmM/l/3o165LMcs2hYE3etp3bcMXwIZQIrvjPDB5x4zyz1DgUrFk4YJfufP1jfQH42q2TeHthdcYVmeWTQ8GajfMO3YXD+/dgUfVqRt3kxnlmaXAoWLNRUiIuO2UQPbdqy+TZi/nJvdOyLsksdxwK1qx0afffxnnjnpvF3yfOzroks1xxKFizM7BnF358/B4AfPfOKbzynhvnmTUWh4I1S8P37sWJQ7dn5ep1nHtjFUtXrs66JLNccChYsySJn31qT/pv25E35rtxnlljcShYs9W2vJRrRgylQ+syHpjyHn96ambWJZk1e6mFgqQ/SZoraepHvD9C0uTC42lJg9KqxfJr56078OuTk8Z5P39gOuNnLsy4IrPmLc09heuBYzbx/pvAIRExELgEGJtiLZZjxwyo4IsHJo3zRt00gflunGdWb6mFQkQ8Dnzk17aIeDoi1vdDfhbomVYtln/f/nh/9uq9Fe8vWcXoWya6cZ5ZPZVlXUDB2cA/PupNSSOBkQAVFRVUVVXVa5Dq6up6f9aavpEDynh1TglPzVjAt/76GKcN6Fj0GryNWZqKsX1lHgqSDiMJhQM/ap2IGEvh8NKwYcOisrKyXmNVVVVR389a8zCmYj6nX/ccd0xfzif22Y3D+29T1PG9jVmairF9ZTr7SNJA4DrghIhYkGUtlg/79+nON47qB8BXb3HjPLMtlVkoSNoBuBP4XES4Sb41mnMP6cMR/XuwZOUazh1XxcrVbpxnVldpTkm9GXgG6CdptqSzJZ0j6ZzCKj8EugHXSJokaXxatVjLkjTOG0yvrm2Z+s4SLnbjPLM6S+2cQkQM38z7XwS+mNb41rJ1bteKMSMqOXHM09z8/CyG7bgVn6n0BDezzfEVzZZbA7bvzE8KjfO+f9cUXn5vScYVmTV9DgXLtc/u1YuTKnsWGudNYIkb55ltkkPBck0Sl5wwgP7bduTN+cv51u1unGe2KQ4Fy7225aVce3olHVuX8eBL7/HHJ9/MuiSzJsuhYC1C7+7tufTkpOfiz//xMi+4cZ7ZRjkUrMU4ZsC2jDx4Z9auC0aNm8C8pW6cZ7Yhh4K1KN86uh979+7K3KWruODmiaxZuy7rksyaFIeCtShlpSVcddoQundozTNvLOCyh30xvVltDgVrcXp0asOVw4dQIrjm0dd5eNr7WZdk1mQ4FKxF2q9PNy48uj8AX79tErMWuHGeGTgUrAU755Cd+dju27DUjfPMajgUrMWSxK9PHsSO3drx0rtL+PE9L2VdklnmHArWonVu24prRgyldVkJt7zwNrePfzvrkswy5VCwFm+P7TpzyQkDAPjBXVOZ9q4b51nL5VAwA07ZqxenDOvJqjXrOG9clRvnWYvlUDAr+MkJA9i9ohMzF1TzzdtedOM8a5EcCmYFbVqVMub0oXRsU8ZD097nD0+8kXVJZkXnUDCrZcdu7flNoXHeLx98hefeWJBxRWbF5VAw28BRe2zLlw9JGuedf/NE5i5dmXVJZkXjUDDbiAuP6sc+O3Vl3tJVfOUmN86zlsOhYLYRZaUlXHnaELbu2Jrn3lzIpQ+9knVJZkXhUDD7CD06tuGq4UMoLRG/f+wNHnrpvaxLMkudQ8FsE/bZuRvfPqYfAN+4/UXeWrA844rM0uVQMNuMLx20M0fvkTTOO+fGCW6cZ7nmUDDbDElcevIgendrx/Q5S/jh3VOzLsksNQ4Fszro1KYV14yopHVZCbeNn81tL7hxnuWTQ8GsjnbfrhM//VTSOO+iu6fy0ruLM67IrPE5FMy2wMnDenHqXr1YtWYd5944gcUr3DjP8sWhYLaFfnz8HuyxXSdmLazmm7e7cZ7li0PBbAu1aVXKmBGVdGpTxsPT3uf3j7txnuVHaqEg6U+S5kra6FQNJa6QNEPSZElD06rFrLHt0K0dl50yGIBfPfgyz7pxnuVEmnsK1wPHbOL9jwO7Fh4jgTEp1mLW6I7cfRvOPbQP6wLOv2kic5e4cZ41f2Vp/eKIeFxS702scgLwl0gOyD4rqYukioiYk1ZNZo3tGx/ry6RZi3jmjQWcf9NEvrxnKQuWrcq6LMupZR+m35gxtVCog+2B2pO9ZxeWORSs2SgrLeGK4UP4xBVP8PzMhTw/E7j3X1mXZTm1a9dWHLJfumNkGQrayLKNTuOQNJLkEBMVFRVUVVXVa8Dq6up6f9ZsU762V3uuHr+WxSvXbnTDNmsMbUrXpf43LMtQmA30qvW6J/DuxlaMiLHAWIBhw4ZFZWVlvQasqqqivp8125RKYPhR3sYsXcXYvrKcknoP8PnCLKR9gcU+n2Bmlq3U9hQk3QwcCnSXNBv4EdAKICKuBR4AjgVmANXAWWnVYmZmdZPm7KPhm3k/gFFpjW9mZlvOVzSbmVkNh4KZmdVwKJiZWQ2HgpmZ1XAomJlZDTW3XvCS5gFv1fPj3YH5jViO2Ya8jVmaGrJ97RgRW29upWYXCg0haXxEDMu6Dssvb2OWpmJsXz58ZGZmNRwKZmZWo6WFwtisC7Dc8zZmaUp9+2pR5xTMzGzTWtqegpmZbYJDwczMamR5kx2zXJC0FbAdsAKYGRHp30jXWgRJPYAD+O/2NRUYn+Y2lttzCpJ6AqcCB/F//0HvB/7h/3GtISR1Jmn9PhwoB+YBbYBtgGeBayLikewqtOZM0mHAd4CuwERgLsn21RfoA/wN+E1ELGn0sfMYCpL+DGwP3AeM5//+gx5GcvfE70TE45kVac2apIeBvwD3RsSiDd6rBD4HTImIP2ZRnzVvki4FroyIWRt5rww4DiiNiDsafeychsKAiJi6iffLgR0iYkYRyzIza/JyGQpmaZO0Q+Hp2oh4J9NiLHckfb7wdEVE3F7MsXN5olnSI0AACyPipKzrsVy6ofBzAeBtzBrbToWfS4s9cC73FCTtWHi6NiJmZ1qMmVk9SCoFLoiI3xZz3FxepxARbxUeDgRLlaSTJXUsPP+BpDslDc26Lmv+ImItcHyxx81lKKwn6URJr0laLGmJpKWSGn0Kl7VoF0XEUkkHAkeTHFYak3FNlh9PS7pK0kGShq5/pDlgLg8frSdpBvDJiJiedS2WT5ImRsQQST8nmYJ60/plWddmzV/h/OiGIiIOT23MnIfCUxFxQNZ1WH5Jug94BziS5PqXFcDzETEo08LM6invoXA5sC1wF7Bq/fKIuDOzoixXJLUDjiHZS3hNUgWwZ0Q8lHFplhOSPgHsQXIBLgAR8ZO0xsvllNRaOgHVwFG1lgXgULAGk1RCslcwYP2yiJgDzMmuKssTSdcC7Ug6MVxHMv35+VTHzPOeglnaJI0DvruxdgRmDSVpckQMrPWzA3BnRBy12Q/XU673FCS1Ac7m/+96fSGzoixvKoCXJD0PLF+/MCKKPpXQcmlF4We1pO1ILpbcaRPrN1iuQwH4K/AyyVTBnwAjAM9EssZ0cdYFWK7dJ6kLcCkwgeTw93VpDpjrw0e1pguu3/VqBfwzzelc1vIUrqDfNSL+VTjxXBoRRW9PYPkmqTXQJiIWpzlOri9eA1YXfi6SNADoDPTOrhzLG0lfIult//vCou1JZruZNZikdpIukvSHiFgF9JB0XJpj5j0UxhbuinURcA8wDfhltiVZzowiuTPWEoCIeA3okWlFlid/JplOv1/h9Wzgp2kOmOte1AV2AAANUUlEQVRzChGx/tjbY8DOWdZiubUqIj6UBNTcACW/x2St2PpExGclDQeIiBVav7GlJNd7CpK6SbpS0gRJVZJ+J6lb1nVZrjwm6XtAW0kfA24H7s24JsuPDyW1pfBFQ1Ifal2Im4a8n2h+GHgcuLGwaARwaEQcmV1VlieFC9jOJrlAUiQTGf6QbVWWF4UvGj8AdgceIjlUeWZEPJramDkPhaqIqNxg2fiIGJZVTZYvkkZHxOWbW2a2JSQdEBFPFWYcdQD2JfnS8WxEzE9z7FwfPgIekXSqpJLC4xTg/qyLslw5YyPLzix2EZY7VxR+PhMRCyLi/oi4L+1AgPzvKSwF2gPrCotK+O9VpxERnTIpzJq9wom/04CDSA5RrteR5I5/PkRp9SbpWZILbY8Fbt3w/Yi4IK2x8z77qGPWNVhuPU3S+K478Jtay5cCkzOpyPLkOJJ27IcDVcUcONd7CgCF6xR25f/2Pnr8oz9hVjeFe+j+03sFlhZJgyLixWKOmes9BUlfBEYDPYFJJCdrniFJX7MGiYi1kqoldU679YC1LJJ+RDINdRngUGhEo4G9SM7YHyapP25gZo1rJTClMP25dpfU1I75Wosws/BzxaZWSkPeQ2FlRKyUhKTWEfGypH5ZF2W5cj+e0WaNLCJuyGrsvIfC7ELb2buAhyV9ALybcU2WIxFxg6RyoG9h0SsRsXpTnzFrynJ/onk9SYeQdEl9MCI+zLoeywdJhwI3kOzuC+gFnOHJDNZc5TIUJHWKiCWSum7k7QCWRMTaYtdl+SOpCjgtIl4pvO4L3LzhlfRmzUVeDx/dRDLPt4okBNZ3FVz/vEOhP/n3MqrP8qPV+kAAiIhXCzdzMmt0ks4juSXnHRGxJpUx8rinsDmF+eVTI2K3rGux5k3Sn0i+bPy1sGgEUBYRZ2VXleWVpFFAf2DHtO4DnstQkLRD4enaiHgn02Is1woNy0YBB5LshT4OXFO4S5ZZs5PXUHik8HRBRJyUaTGWe4XZR7uR9Nh6xRMZrKEkfb7wdEVE3F7MsXN5TiEiDsu6BmsZJH0CuBZ4nWRPYSdJX46If2RbmTVzOxV+Liv2wLncUzArFkkvA8dFxIzC6z7A/RHRP9vKzOon7/dTMEvb3PWBUPAGMDerYixfJP1KUidJrST9W9J8SaenOaZDwaxhXpL0gKQzJZ1Bcn/mFySdKOnErIuzZu+oiFhCMsV+NsmV8xemOWAuzyl8FEkVwELPDLFG1AZ4Hzik8Hoe0BX4JMlU1TszqsvyYf01L8eSXBS5UNKm1m+wFhUKJHPJ+0i6IyK+mXUx1vxt7HoESeWegWSN5J7CeasVwHmStibpzJuaFneiWUnM7h4RL2VdizV/kh4FzoyImYXXewHXRcSgLOuy5k9SCck9YKZTaM0jqT3QMSLeS23clhYKZo1J0tHA5SQ3Wt+eZDf/7IiYkGlhlguSnomI/Yo6Zh5DQdKbJMdz50XEPlnXY/lW6JT6MDAfGJLmtzhrWSRdTHLP7zujSH+scxkKZsUi6SLgFGAkMBD4GvCNiPCNd6zBJC0F2gNrSc4rCIiI6JTWmLk+0Szp4I0td697a0Tdgb0jYgXwjKQHgevw3disEUREx2KPmes9BUn31nrZBtgbqIqIwzMqyXJKUvuIWL75Nc3qrjAxZgSwU0RcIqkXUBERz6c1Zq4vXouIT9Z6fAwYQDKn3KxRSNpP0jSSGSJIGiTpmozLsvy4BtgPOK3wehlwdZoD5joUNmI2STCYNZbfAUeT3PiEiHgR2OhhS7N62CciRlG4NiEiPgDK0xww7+cUriSZhQRJAA4GXsyuIsujiHh7g6tMfatXayyrCzcFC4DCxWvr0hww16EAjK/1fA3JZeJPZVWM5dLbkvYHonBfhQsoHEoyawRXAH8Hekj6GXAScFGaA+b6RLNZ2iR1J7l47UiS6YIPAaMjYkGmhVluSOoPHEGyff07IlL90pHLUCjceS1Imt/5zmtm1ixJ+mtEfG5zyxpTXg8fnVn46WO7lgpJPyL54rEsIi7Luh7LrT1qvyicX6hMc8C8hsKszV0SLknFumzccmlm4eeKLIuwfJL0XeB7QFtJS9YvBj4ExqY6dh7/LhY6V94B3B0Rs2otLwcOBM4AHomI6zMp0MxsMwpdUq+LiC8UddychkIb4AsUrgQEFpFc0VxKciLw6oiYlF2FZmabJ6kqIlI9XPT/xsxjKNQmqRVJf5oVEbEo63rMzOpK0tXA9RHxQtHGzHsomJk1V4UWKn2Bt4Dl/LdL6sDUxnQomDUeSeeRtLy4IyLWZF2PNW+SdtzY8oh4K60xW1rvI7O0iWQyw51ZF2LNX+GPfxfgk4VHlzQDAbynYGbWZEkaDXyJ/37J+DQwNiKuTG1Mh4LZlpP0+cLTFRFxe6bFWG5Jmgzst/5eHZLaA8+keU4hrxevmaVtp8LPpZlWYXkn/m9nhrWFZalxKJjVQ0RcnHUN1iL8GXhO0t9JwuAE4I9pDujDR2YNUOhv/yWgN7W+ZBX7KlTLL0lDSSYvADwRERPTHM97CmYNczfwBPAv3IDR0iOSm+ukeugIvKdg1iCSJkXE4KzrsHyS9EPgZJJebgI+BdweET9NbUyHgln9Sfop8HREPJB1LZY/kqYDQyJiZeF1W2BCROyW1pi+eM2sYUYD90laIWmJpKW1Wh2bNdRMkmae67UGXk9zQO8pmJk1UZLuAvYCHia5qdPHgCeBuQARcUGjj+lQMGsYSVsBu1LrG11EPJ5dRZYXks7Y1PsRcUOjj+lQMKs/SV8kOYTUE5gE7EtyxenhmRZmuSCpR0TM3WBZv4h4Ja0xfU7BrGFGk+zevxURhwFDgHnZlmQ58oSkU9a/kPQN4O9pDujrFMwaZmVErJSEpNYR8bKkflkXZblxKDBW0snANsB0YO80B/SeglnDzJbUBbgLeFjS3cC7GddkORERc4AHgf1Irpr/S0QsS3NMn1MwaySSDgE6Aw9GxIdZ12PNn6SHgTnABSTnrf4EPB4R30xtTIeC2ZaT1CkilkjqurH3I2JhsWuy/JH0qYi4q9brMuC7EXFJamM6FMy2nKT7IuI4SW+SzB+v3ZMmImLnjEqznCncknPXiPhX4YrmsohIrWW7Q8HMrImS9CVgJNA1IvpI2hW4NiKOSGtMn2g2awBJBxTuhoWk0yVdJmmHrOuy3BgFHAAsAYiI14AeaQ7oUDBrmDFAtaRBwLeAt4C/ZluS5ciq2pMWCucUUj2841Awa5g1kRyDPQG4PCIuBzpmXJPlx2OSvge0lfQx4Hbg3jQH9DkFswaQ9BjJPPKzgINJrmaeFBF7ZlqY5YKkEuBs4CiSyQz/BK6LFP9wOxTMGkDStsBpwAsR8UThfMKhEfGXjEszqxeHgplZEyPpEZJzBwsj4qSiju1QMKs/SfsCVwK7AeVAKbAsIjpnWpg1a4VrEwDWRsTsYo7thnhmDXMVcCrJCcBhwOdJ7q1g1hCzNnfeQJLSOLfg2UdmDRQRM4DSiFgbEX8m6Wxp1hCPSPrKhte8SCqXdLikG4BN3oCnvrynYNYw1ZLKgUmSfkXSvKx9xjVZ83cM8AXgZkk7AYtI7uxXCjwE/DYiJqUxsM8pmDVA4djv+yTnE75G0iX1msLeg1mDSWoFdAdWRMSi1MdzKJiZ2Xo+fGRWD1lOGTRLk/cUzOohyymDZmlyKJjVQ12mA6Y1ZdAsTZ6SalY/mU0ZNEuT9xTM6kFSG5IpgyOAjU0ZvDqtKYNmaXIomDVQsacMmqXJoWBmZjV8TsHMzGo4FMzMrIZDwVo0Sd+X9JKkyZImSdonxbEelTQsrd9v1hh8RbO1WJL2A44DhkbEKkndSXoYmbVY3lOwlqwCmB8RqwAiYn5EvCvph5JekDRV0lhJgppv+r+V9Lik6ZL2knSnpNck/bSwTm9JL0u6obD38TdJ7TYcWNJRkp6RNEHS7ZI6FJb/QtK0wmd/XcR/CzPAoWAt20NAL0mvSrpG0iGF5VdFxF4RMQBoS7I3sd6HEXEwcC1wNzAKGACcKalbYZ1+wNiIGAgsAc6rPWhhj+QHwJERMRQYD3xdUlfg08Aehc/+NIX/ZrNNcihYixURy4BKYCQwD7hV0pnAYZKekzQFOBzYo9bH7in8nAK8FBFzCnsabwC9Cu+9HRFPFZ7fCBy4wdD7ArsDT0maRHLl844kAbISuE7SiUB1o/3HmtWRzylYixYRa4FHgUcLIfBlYCAwLCLelvRjkiuV11tV+Lmu1vP1r9f//7ThxT8bvhbwcEQM37AeSXsDR5Dc4vN8klAyKxrvKViLJamfpNr3Ux4MvFJ4Pr9wnL8+bbF3KJzEBhgOPLnB+88CB0japVBHO0l9C+N1jogHgK8W6jErKu8pWEvWAbhSUhdgDTCD5FDSIpLDQzOBF+rxe6cDZ0j6PfAaMKb2mxExr3CY6mZJrQuLfwAsBe4u9FUSyZ3czIrKbS7MGpGk3sB9hZPUZs2ODx+ZmVkN7ymYmVkN7ymYmVkNh4KZmdVwKJiZWQ2HgpmZ1XAomJlZDYeCmZnV+F/CfifB/QM9vQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "bi_fdist.plot(3, cumulative=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Stemming"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "re-test re-testing re-test\n"
     ]
    }
   ],
   "source": [
    "from nltk.stem import (PorterStemmer, LancasterStemmer)\n",
    "from nltk.stem.snowball import SnowballStemmer # This is \"Porter 2\" and is considered the optimal stemmer.\n",
    "\n",
    "porter = PorterStemmer()\n",
    "lancaster = LancasterStemmer()\n",
    "snowball = SnowballStemmer(\"english\")\n",
    "\n",
    "print(porter.stem('Re-testing'), lancaster.stem('Re-testing'), snowball.stem('Re-testing'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('arabic',\n",
       " 'danish',\n",
       " 'dutch',\n",
       " 'english',\n",
       " 'finnish',\n",
       " 'french',\n",
       " 'german',\n",
       " 'hungarian',\n",
       " 'italian',\n",
       " 'norwegian',\n",
       " 'porter',\n",
       " 'portuguese',\n",
       " 'romanian',\n",
       " 'russian',\n",
       " 'spanish',\n",
       " 'swedish')"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Fun fact: SnowballStemmer can stem several other languages beside English.\n",
    "# To make, for instance, a French stemmer, we can do the following: french_stemmer = SnowballStemmer('french')\n",
    "SnowballStemmer.languages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{33: None,\n",
       " 34: None,\n",
       " 35: None,\n",
       " 36: None,\n",
       " 37: None,\n",
       " 38: None,\n",
       " 39: None,\n",
       " 40: None,\n",
       " 41: None,\n",
       " 42: None,\n",
       " 43: None,\n",
       " 44: None,\n",
       " 45: None,\n",
       " 46: None,\n",
       " 47: None,\n",
       " 58: None,\n",
       " 59: None,\n",
       " 60: None,\n",
       " 61: None,\n",
       " 62: None,\n",
       " 63: None,\n",
       " 64: None,\n",
       " 91: None,\n",
       " 92: None,\n",
       " 93: None,\n",
       " 94: None,\n",
       " 95: None,\n",
       " 96: None,\n",
       " 123: None,\n",
       " 124: None,\n",
       " 125: None,\n",
       " 126: None}"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from nltk import word_tokenize\n",
    "\n",
    "sentence = \"So, we'll go no more a-roving. So late into the night, Though the heart be still as loving, And the moon be still as bright.\"\n",
    "\n",
    "# This uses the 3-argument version of str.maketrans with arguments (x, y, z) where 'x' and 'y' must be equal-length strings and characters in 'x' are replaced by characters in 'y'. 'z' is a string (string.punctuation here) where each character in the string is mapped to None\n",
    "translator = str.maketrans('', '', string.punctuation)\n",
    "translator\n",
    "\n",
    "# This is an alternative that creates a dictionary mapping of every character from string.punctuation to None (this will also work but creates a whole dictionary so is slower)\n",
    "#translator = str.maketrans(dict.fromkeys(string.punctuation))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['So', 'well', 'go']"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokens = word_tokenize(sentence.translate(translator))\n",
    "tokens[:3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['So', 'well', 'go', 'no', 'more', 'arov', 'So', 'late', 'into', 'the', 'night', 'though', 'the', 'heart', 'be', 'still', 'as', 'love', 'and', 'the', 'moon', 'be', 'still', 'as', 'bright']\n",
      "['so', 'wel', 'go', 'no', 'mor', 'arov', 'so', 'lat', 'into', 'the', 'night', 'though', 'the', 'heart', 'be', 'stil', 'as', 'lov', 'and', 'the', 'moon', 'be', 'stil', 'as', 'bright']\n",
      "['so', 'well', 'go', 'no', 'more', 'arov', 'so', 'late', 'into', 'the', 'night', 'though', 'the', 'heart', 'be', 'still', 'as', 'love', 'and', 'the', 'moon', 'be', 'still', 'as', 'bright']\n"
     ]
    }
   ],
   "source": [
    "for stemmer in [porter, lancaster, snowball]:\n",
    "    print([stemmer.stem(t) for t in tokens])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Lemmatizing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lemmatization aims to achieve a similar base \"stem\" for a word, but aims to derive the genuine dictionary root word, not just a trunctated version of the word."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "brightening box\n"
     ]
    }
   ],
   "source": [
    "# The default lemmatization method with the Python NLTK is the WordNet lemmatizer.\n",
    "from nltk import WordNetLemmatizer\n",
    "\n",
    "wnl = WordNetLemmatizer()\n",
    "\n",
    "print(wnl.lemmatize('brightening'), wnl.lemmatize('boxes'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'brighten'"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# As we saw above, sometimes, if we try to lemmatize a word, it will end up with the same word. This is because the default part of speech is nouns.\n",
    "wnl.lemmatize('brightening', pos='v')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}