01_week_01_sos.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "SoS"
   },
   "source": [
    "# Data exploration at the shell\n",
    "\n",
    "**SERHAT ÇEVİKEL**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "SoS"
   },
   "source": [
    "In the first part of this session, we will use the shell to explore and wrangle our data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "SoS"
   },
   "source": [
    "## Some notifications"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "SoS"
   },
   "source": [
    "### The Moore Foundation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "SoS"
   },
   "source": [
    "This facility, known as \"Binder\" enables us to create \"stateless\" and \"zero dependency\" teaching environments to which you can access anywhere from the web. All it needs is to click a link! And each click starts a new instance of a Docker image at the same state (programs, configurations, options, data, files, material, etc.)\n",
    "\n",
    "\n",
    "As Binder faq states:\n",
    "\n",
    ">\n",
    "How can mybinder.org be free to use?\n",
    "The Binder project has a grant from the Moore Foundation to sustain the cloud resources running mybinder.org. In the future we hope to see more public BinderHub services running that can form a collection of community resources for interactive cloud computing.\n",
    "\n",
    "\n",
    "[Gordon and Betty Moore Foundation](https://www.moore.org/) web site says:\n",
    "> Gordon and Betty Moore established the foundation to create positive outcomes for future generations. In pursuit of that vision, we foster path-breaking scientific discovery, environmental conservation, patient care improvements and preservation of the special character of the San Francisco Bay Area. \n",
    "\n",
    "**SO SPECIAL THANKS GO TO GORDON AND BETTY MOORE FOUNDATION FOR ENABLING SUCH A FACILITY**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "SoS"
   },
   "source": [
    "### Other acknowledgements"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "SoS"
   },
   "source": [
    "**Thanks also go to Google Cloud and OVH for sponsoring our computers on the cloud for mybinder.org**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "SoS"
   },
   "source": [
    "**And thanks also go to Deutsche Forschungsgemeinschaft (DFG) and Leibniz Institute for the Social Sciences (gesis) for powering the computers on the cloud for notebooks.gesis.org:**"
   ]
  },
  {
   "attachments": {
    "image.png": {
     "image/png": "iVBORw0KGgoAAAANSUhEUgAAAKMAAAA8CAIAAACFPp1BAAAYZUlEQVR4Ae1cC1xTR9afBBIghCRIHpDQGFqECvQTKCCPuIACivhAqRaL69rWSq2obNGuW6v1/ejWWttut2q3uPXVFp9AfVZUQGmLVawKCJFHQAIGSEISQnKTzPeFkfGK9FvdLV11c3/+8Ny5Z86cOWdeOfOfoUAITSaTvrvblcGg0+ngP/SsXr0uIiJs3Lix/6Hyn/xiqVkLFj39jP/EiamjRsWHhoZnvv6GwdDz79TbarW2tbVZLJaHErJn796KiisPlcXO/FAWoJ49ey46KvJvf/tk8+b3MmZmnDx5yoMrkMvlDyWFzPzV19+kvfCig4MDOfGf0kKhkOHK+KdsdoZ/2QJUOo02MnLkc0FBUmlMzpvZ9XW1oaEhiUnJZIlyubygoFClVpMTB6RVarWs9qZSqWxv79DpdIjn6rVrBQWFNTU15CxtbW3Hj58oLT2PhhCCIHwkEgDA8eMnKioqyJwAAKvVWlBQeH96Pzb76/9ngYiI6FWr1kLSYzQaXRiskpJSlPbO8ndF3pIYaazIW5KbuxMlzs2c987yd3GmrR99nDFzFoRwyZKlIm+JxMc3JCRsz959EMK5mfP8/APi4hMCg0ZcuFCGsmRn50h8fOPiE/z8AzZs2AQhnD371bmZ82bPfjVGGivx8Z09+1Us/PTpIqRAYNAIcjpmsBMPYgFwv6chhCEhYciRubk7KRTqj+XlEML9+w9QKNSfr16FEMbFJyDXojKWLFkaGDQCQtja2rpq1dqQkLAbN25otdrc3J1sDpcgCAhhU1OzUtkOIczOzqFQqMjrWq22sbERQjhtWjqFQj127DiEsLGxkUKhoobS1NRMoVDf37wFQqhQtFIoVNQyHqRudh6yBagD9nc2h6PT2sbeVavX5uXlhYeFAQDS0qZOnjzlg80fAgA4bDaHw8F5mUwmehUIBGKxN41G9/PzYzKZdCcnk8mkVCoBAN7eIi7XQ6FQbN26pbi4OCoqEgDAZDLFYjEAQCaTvTY3E629xWLxSxkzjx49BgBYvvzd8eMn5LyZDQDw9BTk5eVt274Dl2snHtwCjgOyatRq76e8AQCurq67d+89eeo7xHbi5KnR8XEAABqN1i8jTjGaTPjTSzPSy3+8+IzvsxNSxu/4fBubxTpz9txQydNSaQzmQQSbw4mKHIkTXV1dTUabHMJM1MpkWQsWEQQBAKi4fEWlUkMIKRQKZrYTD2KBATyt0+kqKi7948tcm6EJgi/g+0gkJpPN0GvWrBoz2uZpZHdcgCvTFdNGoxHTAIAtW96fPv2F7OwcP7+A6uprngIB+SumaTQauYkAAOhOth/3ep2ew+Z4i0RIAR+JJDh4hN3N2G4PTtg8Taff00HHjBk7VPL0c0FBAIC2tttTUiffH9DQd3eTy2huau7W302h3SswKiryhx/OS0fFvfnHxbm5Oxob6q5eu4bkYyH9mg45fehQ8dKlb+EUO/GvWYCq79Z3dqrMZnNra9uBAwdDQ8Nv1NScO3saiZv72pzx48crFAr0Wl/fgH47SWNivvhiJ0qsqqreunULj8dFr1wut7ZWhui2tjaz2Yzobr2+d8KmTJ48ZeTIGPybDTEQBEEeDPS9DwDg3ZXL9+//5sCBg0hIa2tbc/MtRNv/PpQFHIf5+n7y10/LL17UqNUmgoiJif7++/M4LPreextqa2uDQ8KjoyKV7e0AgIKCwwCAt9/+08lTp3yeHjZs2DBXBmPy5Cl4gZacPG7ZsuX+zwb+eelbHh4ei5f8KWD48MbGRhNBfPzxVgDAgQPfjEkY++yzQdFRkY2NjdOnT7u/y9Ic7wwz4WFhH27d+vtZL2/b/jmdRqu48vPWDzd7e099qEramQEAlNbW1rq6eovF4unp6ev7zIBGKS09f+nyZZab2+jR8WipjNj27vtKoWidmTFDIBDU1zf4+NhCHwAAlVq9a9fu8LCw8PCw7747XVNby3Jzmz37D+gr+ltUdOba9etOTk5JiYk+PhK5XO7GYrn3refRKOLl5YWY5XL54SP5AICoqMjw58OAfTVGNuWD0RQI4YNx2rkebwsM/Hv68a6TXfuBLGD39EBWeRLT7J5+Er06UJ3snh7IKk9imt3TT6BXZbKbOIyBq0e1WCzt7R3NzbcUCkVz861/E3CC5ZKJtra2zNff+G+IeCgUitWr1+GgENkID0sbDD2Zr79RVVX9UBlrampCQ8NTp6Rdvm+PH2TMnCXx8U1MSo6Rxvr5B+zff4C81fWr0HV19XyBqLKy6leR9igLqays8vMPQPuw/6aeWq2WLxBhlMADSktMSo6LT2hqatZqtWinuKmpGeV1PHu2+KUZ6QsXZhG2x+TpOfAOxEO1rPuZRUIvvNl1/9cnJmX48GdvVF//taojGTr0YUFaGrVmw4a13t4ipMOCBYuCngtas3olAMDBy0s4dcrk3/1ulLs7x8PDw8nJGStaUFB49tw5lpubh4cHSjQYehwcqBQKpaCg0GDo9vT0BADs3ffVTz/95OTkjEPfAICiojMnT33XrmwXi8UEYcrLO7B48R9lMtn+AwednJwEfD4WaDT24OCr2WzW6bRIh/b2DgaDoenq+sc/vlS0KPz9/bBiAACdTrd3776KiivBwcEAgK4uDcrV3Hxr31dfNzY08vl8BuMOMK2qqvrgoUPtynZfX18AgMlkwhYsKCgsKyvz9fVFOkAI1RqNi7OzTHYzb/8BXHdUncCAQBrtnt2/vfu+su3WPGfbDQIAQAgJgqBSbSZC+pvN5i+/3HXzZl1AwHCy/gcOHCwpLXV0cEA2RJ9KS88fO35C0aIQiYQODo47d365YEFWZ6dq31dfEybiqd59ZMRpsVhOnjx1uugMlqBSq29UVxcWHvXgevB5PAdHx9ra2m/y9ru6Mp7yFjGZTDDgiF1XV+/nHxAREZ2YlCzylszPWohGgPz8gmnT0rOzc1wYrNzcnUple0hIWIw0NkYam5IySaOx7RwrFK0hIWGBQSMSk5L9/ANKSkqtVmtqalp2dk5KyqS4+AQXBgvjRt7fvCU1NQ0PTRculMVIYy0WC4RwftbCjJmzUlPT4uIT0PyC2fLzC1wYrLj4hLj4BDTvIIBKbu5OkbckMSk5MGgEhkwtWbLUhcFCbDHS2MSk5Lq6egih0WiMkcYGBo2Ii0/gC0QI7mKxWOZmzsuYOQsNg2wO99ix4xs2bELVFHlLLl++jNS4ceOGxMcX1d3PPwANkpWVVSEhYYjOzs6ZNi09Y+YshKyKiIhGI6pC0RojjY2IiEb6I2atVotmT2Q0NIempqbNz1qILMDmcJcsWYqK1mq1qalpyDtsDhehg06fLgoJCZP4+AYGjcjOzrlwoSwlZRJ6jZHG/nz1qg1dNDdz3rFjx/PzC/LzC7q7DRBCkbckJWUSkltZWYXxPSUlpRQKNWPmrE6VCkKYkjIpMSkZsXWqVEaj0WK1IFsjRBFBEFqttrvbwOZwQ0LCUK5jx45TKFRUw1Wr1sZIY5EECGFJSamff4DZbIYQpqamYWATgrah9tGpUlEoVIxi2/rRxxQKNT+/AELowmAhAkKIysrPL8BIJghhYlIyhUJFIKfU1DQ//wBU9IYNm9gcLqq7n3+AC4PV0tICIdyzdx+FQk1NTUPVQfZFWUTeEgxqi4tPiIiIhhBWVlaJvCWoJc3PWkihUPFEyxeI5mbOgxBmzJyFkFgQQmQcBOcKDBqBmgJKhxBKfHxF3hJkqMuXL1MoVNTOtFotRuTV1spcGKzLly9brVbUXHJzdxqNRrPZbDQaU1ImvbP83e5ug20pnpiUzOZw4+ITQkLC/PwDlMr2kpJSNoeLrQ8h3PrRx3yBCEL4Y3k5hUJFRkRoQJG3BL9CCD/btt2FwSLnhRAqle1sDhcrhyqGeuH7m7fgJgUhvHChLCIiGnkaNUEsan7WQsT5/uYtIm8JTkftEkEZJT6+2dk55E9x8QnkMaO2VsbmcLVarVqjQTXCzH7+AUgliY/vZ9u2o3SCINgcLu7HqN8gSJ3ExxfnhRAisUplO16RTZuWTq7ahg2bUGtYtWotm8NF/kMSUHMkS4MQms1mNodLXiDHxSfgsRAxo6YZEhKGFY6LTyBnSU1Nw2ObY0tLy6ZN6zPnvoZnkV2794SE2CY//MTHx61btxEAYOwxBgQ+h3ecPv7oQ1Wnysdn2IsvTtv22acAgNPfFSH4Ec4LAOju1vv7+YlEQpzIsIGH7oGm4E+YYHPYfn5352aaIw3NuzU1Nf3mvOAR/9Pa2gYAOHQwb8LEKYePHFm3bs1LM9IBAC0tLRMmpGCZPD7Pv1dmW2ur0WicPn0GQkDQaLTammqFohUAwOfxuX3rko6ODn8/vyFDhiAJer2e4WpD17R3dKhUapydIIguTWdnZyePd2f9gTAzkl5cM8rLcGWwOWwAwIoVy2613PLzD5g6NfWLv++g0+mF3x6NjhmFlUSEwWDw9/N75pmncbpr37IDALBt+44v/r6T4cqg0WjVN2qEfZt+tm1+EroLwbOQBNv6wulhDunQSQgyOp3+zTf7jh8/Me+NrNDQ8EuXyt2HuHffC0fBilosVkz3GsLJpse9ajn2rncQeIhcEFljG92LKcPSMAAmODi4ual+48b3ZmZkyGpvrlixzJXB6FcEQZgcHByNJpOTk5N0VAxB3AFKpE6ZHB8X2yvcpNVqkXAIAUGYELAJLeVQuk6nd3fnjEkYrdPpUcrLL/9BKBQ2NTVjrfrpiQuy+emzT38/M2Phwmz/ZwPrbtaIhKLr1yvJGTFN9OE4bKUTBK3XU8tXrFy3ds369RtSJoxnujLT0qZ1dHTgLL9E2DzdrxUkJSbs2rWHnOHkyTuIQZ1OZ7rXygCAcePG1tfVMlzZx4+fmDPnldjYMeS8AADk4186v6NU2gAO6JE3yhsaG6lUW+Tu/oIQT2zs777+Oq8vh+3/H374MT19Ok5ZuvStMQmjR0aMXLFiWXRMdOG3RzHSoexCWUXFJRrNUSwWa7XaBQuy/rWdbi8vz+5uA3kgRKVrNP/88APilEpjLl0qF3h6b9/x+SuvzF65coXZbHZ0vGdhDwAw9zVElIvZi9fbvHnL7j170KBlg9r9QtdCTQ2PBFS1WoMbJhKXljZVIODHj05Er+UXLy5ZnLNs2VKbUL1erdag9F7o7k0UU2tuvtVj0On1+vCwMDc3N+moOJRuMpkMhh4ajabWqK3Wuye11Go1Gr3HjUsqL/++tPQ8AEAuly/KzkG/VXrH/G59X49B0EE0Wrw0I93dnRM/OrGtrU0mu/nyy3N6DDquhweEUCa7iXQrLi51dmHaQMTvvH3hfMnGje9purqKis4syn6Tx/dqb+9gs1hSaYy7Ow/Fs3Q6XfnFi2irXqVW46YPoVWlVuMhxGgyqXsPssxIf9FoNE6YMBkVJ5fL0fkSB0dHtVqDWrZep+823MXW2V57XVJf39DVZbOhSq3WarU6nd7bWxQcHDrMbzhSxmKx6HQ6BwdHtUZN7h6aPk8JBPxz54pR0Zs/+LC2phr/DO7W68nTor67+9Kly4jTUSQUopaC3tHfkuKz45JTQkPDuTzelSs/L1781sIFWb3YQrpIeHe63b798yP5+cN8fSuu/PzG/KypU6f0AnXLE5OSwyMiJUOH1spkO3P/PnJkBI/Ho9HuHuQUCoV0J9voHRwcvOyd5Uljx48cGQEASE2dpLbVv4vFYvO4XI67bWJDD0YvAQBKis+mpU2fNGkqQZhmv/wHvX56e+/wNfvlVxFzba1sVy+2VSAQHD16dFH2m4XfHqXRaH9e+taRIwWtrQpPT8GRwwcTk8ZFRsYM8/Wtlcni4mLDnn8eACASidzc3JAcKpUqEolwzMeNyRT2Vp9CoZSUnJk0aUpoaLhQKKy48nPW/HnBwcFOdDqOEXE4HDaLpL+7rUYAgIOHDn/yyV8DAwKuV1ZOnZq6IOsNAEBx8ZkxY8ZiZd7/y6aJEyfweDwnZ5uV0CO0RZ9snX7XlzvTZ8yUjoqj0WjhYWEvvDBd23cwisvjYeUBAH96a/HUtOmhoeHr16+ldKpUzk7OLi53AyZ9km1HpDpVqqjISAwbQudv8YrMYOgpKyuTy+XBwSNQBAPnLSo6I5fLxWKxVBpDp9NVajXLzQ3HK1RqNfkQb4XtuRIbG+vjI1Gp1W5MpqOjY1eXhkZzwoohpKItAtD3VFVVu7u7e3oKQkPDFy/JeWlGenPzreKSEgDApIkTyJwAAIRGtVqtXkLx9WtXuNw7saDS0vPypibxU09FRUUi9ci6Wa1WTVcX1tw2RPX0sFmsPhVAQUGhVqd77rkgBHW1WCxdWi2bxaJSqf0UNhh6CMLIYrHNZvP33/8gk8n8/f3R8QYsrbT0vEwm8/T0lEpjmEwmNgVi0HR10Wl0ZBCDoefQ4cND3N0RbFfT1YW00nR1uTg740gUAKCmpubiT5dGx8c/fugii8ViMBiwIzd/8OGaNeuam+pxCjYcAAAF9XDNUT84U3SKzPNfQvdfAjwW1X593vyGhkaJZOjt28offyz/at/uAd0MAGhtbU1PzxAKvVyZrrU1MrVGXXyu6LGo46+u5OPXpwEAVVXVR48db25qFgj4c+a8iofi+60DITx//kJR0VmVSvXscP/7V8v3Z3lSUx5LTz+pzhjUetkxJ4Nq3kdIuN3Tj5AzBlUVu6cH1byPkHC7px8hZwyqKnZPD6p5HyHhdk8/Qs4YVFXsnh5U8z5Cwu2efoScMaiqPJbR0EG1yGAIV6nVPQYD2ncny4cQmM1mPp+HI/Poq9lsbm/vgNB6fxbEYLVaaTT6gMFBlVptMVsIwnYfEJVKZbHYaFPEHiMjW36w6MzX39ixfZtQZLuPq99DEASPxw0OHrFo0QJ0Gxja+I+RxhqNxl+K53d2doaGhpSWnMXSrl67tuvLPRfKypRKpQ0l0wthoNEcXV1deTzeju2f2fs0ttUgEnq9DYTU2dnZrwzUlevq6iuvX927Z3dxcTG6v8tqtWi12h7Dnes4ycAmJKHHoEOYCPS6ceN7b7/9Z0Q7uzCRWFMvcgsJkcvldk/3M/5v9Eqn000mU5emE3V0Go3f2FCXNHZ8ddVVsVhMpTq4ubnR6XTUp2k0x8aGun6aYfAPcrOzC1Mg4KOu3HKr/wW/dDrdPnr3M+CgvNbXN7S2tmJUBSqDMJs///wLPKojd742N3PbZ5/W1zdERknx6N3Z2ZmT88fgEf+DYU8mo9HDw2PixAlVVdWBgYEs9hAmk0mjOXZ3G5S3FbFxo+Pj4gIDh7u6ul6vrDp06PBf3ttoO2Bif/6DFoiIiKZQqCJvichbQqFQ8fWrfIGIzeGKvCV8gYiMse+nKjo/gM4AIM5+mHDEbzQa7aP3oHTifkIrKipu3qzD06d4qBgvvqZMmVxe/j3iZ7GHKJXtZrOZRQKgAQCcXZjbPtseHDwCYxdNJhMSUlNT24sBtWGZlbcVL2XMxEBYsg50Ot3uabJBBotev37T/v3fYOkJiWNPnjiKXsn31zs5OSlvK5qamiWSoZiZRqMJBPzVa9aRF2gAgPHjJxQWHtGoNQgFixrB65lzccZ+hN3T/QwyKK/uQ9wBADy+7Xo15W0Fn88bsBgEQtVo1BTKnZvdEBtBmHvHg7toyR6DDp8kRTxGo5HFHiIS3TlPa0NAr1h56NBhoVDY0tIyfdo0u6cHtPmgJGI0MVk6+WwH6pcCgcBqvee8C7oem3ygt7FBh27QZXPYvX19CBoPbty4gYG8165eq7x+tfL6VQDAteHD7Z4mm/03ovFdmQCAwsJvcalarVYoEnt5eSHoP0onCEJ5W5GXlyeVxuAWYLFYnZ1twG3kV9wIPtiyFV/nK+o9Ly8UiVtuyd2HuNs9je382xF6vb65+RZBEF/k7jx3tqjvJ7Vjj0EXHTUeANDRoeqnjXioWDDQhdlz5ryyY/s2lUrNZDKFIvF3p05kLViUmfka05XZUN+ApnAkyu7pfiYd9FehSHyh7PvIKCnqrNjNbW23AQALF9rOyqCoNVkVbdedQ4HkRABAeFjYypWrV65cAQBwd+cIReJP//rJ7t17nXqPyOBTogAAu6f7me63eMVXXCM3EwSBolp/eX8zioZSqQ7kO7Dvj4aStVyxYplKpdq6dUuXppPFHsLjexl7HxRfQ3P/7bbb9shJv1DEoLyi+x0oFOov/YuLT0DXb6Di0T0UZGZ818Mv6ZefX5CYlIyCJ+SMFAqVLxDZbo6w3wVM7h+DRKMrrll9B/twKSi6GRoaggMp6JNOp8svKMRH27Va7dixSfgObJz9fkIul8tkN1UqldFkOwXu5uYmfuqp559//v8Csf8LJp4/OYMbdr8AAAAASUVORK5CYII="
    }
   },
   "cell_type": "markdown",
   "metadata": {
    "kernel": "SoS"
   },
   "source": [
    "![image.png](attachment:image.png)"
   ]
  },
  {
   "attachments": {
    "image.png": {
     "image/png": "iVBORw0KGgoAAAANSUhEUgAAAHMAAAA8CAYAAABVTYVfAAAgAElEQVR4Ae19C3hU1bXwWnufmclMXiQhBBEwQAjJzJmIplVbrfLXWguE2NZLv7Zeba+ttF7rWwRta+Pfq7a+BdRibbW21iqt1yrUgqjU+tb4yDwgECBACIQ8J5PM65yz1/32kBnOvMIg0Pt//+f5vslee++11l77vfbajwB8+v1/UwKYb04WL36GD84cLLLrYNFjUdQcodhGeyQILS0iXx5Hgte4ZLVlki1qZ4UKl3RiVDcORG3h1kd+oB0Jn1y4Mj+9lb32wpJhix6tRMXWS6PDJVplb2V4zZpvGLnojlt4Swv78kih3aErljAWM8UWoyER1C2OcHhTS4ueT7qHrcymZStORmQ/R4IFgBAvWBPjKJB4dO/gwI2tj7SEmpev/J0pDoSATWvvvPIxc9h48KKW1Q6MaFcA0HcB0JkNl4h2AtHqvp07V7y15r5wNpxcYfNaWpSi8MSrGMDFBHAyImTLPxHQbiJ6mYBe2j/o+0vrI4/kbECLblz5VWRwrjnN539x5VUAQOawXHDT8lXfR4BLEeBzuXAAaB8RvgooXtJHtz314sqV0Wy42TKTxGtavvI1JDwrR6aTeEQQHRzoayivmNieDJS5EcZjL9x59RIAOGzLal6+8qdA+FNAsJh55IYpJnTxzbV3X/1cPgXXtGzFJQzYo/nzP5hyaKD3pI2P3LonVxoXLHvwLkK6wSyn/8/3F3R0dMRy0Ujc+ctXftECsBYA7Wbaw8GhkYGGjatu8cnBKh1XSQ+Q/vlXrihRHHgAAW2JtkskhvVoeIM2PPiuYrWVK0WlX0FujbduRLClV6TkI7SY/cQTTyydPHnycGtra/bWvfgZvqjmwHogODeZFpBuxGKvRHq7/xDo2rxVsdksRdPqTrOXTroIueWUg40LrUzhzy684b7bHLve/NmaNWtyDo2Llj1wByJbnp5XIhEkQQEEYMhYCSAWpeOEDmwrr6qq6u/p6QmNVzlmOqvVWlZdXT3U2dkpe1BGD71g2cpvE+CTZhoJE1CIDGNQwshYMSIrSceJ9HVXVFY6Hb29filPSoVmVKYcipQI8yKALcHI0DVv92trloiR0UhYF2GiSIxzvrrKfc7skhnqi8gwffiNkxqGbi+2FBf39/ePjvXOjIwtmnXgOZQVOfaREAM977+4OLq/e1BwEdY0LSKE0IZ2+ncYhvHM1NMXzi2ZNvuFBD5XLD8eKnO9BrDmVQDIaDDzl644J70itcjIhiHfu78a7m7vIiJdCKEjIhVPqS0rmlY711E59XrG+UkyjWhwpKy4uNjR09MTAYCcDSYhz5hbwRiTU0DGcHj+NfedQJhakXos/P5g+4d3hnb6dsk8REU0Lo+tclpJ2fQ6tXBy9RVMsTZI3kZkZEJhIRRNn94YbW1tHb8yiyMVTyDAtIRwBBTZvfa3SyJGJGwTtv5Y/0hQL9KjiqLQltee6+VvvDr5MxdftT/LfAooDA4FYK0YrmCd0JlgmXQXLH3gKwjQlAggIWKdrz719fBgYFjRxeCIPjIMAOGurq7EMI2h0JpXGr52xbesxaVPJejs5RPXV1dXl3V2do6kD+kWxlYm8KSrhQIv7Nnw1C9I0KjBjICmaSFN0zTGGI1u+3BvwZ7NmyEEz1oqq0pnnnvhGkaKQUQ4b9483LRpk5lVTlghxWa322UDl9NYSgO2Flj+aCbUo+H3d214/JpoNBpCxIAQYiQpT+cWjBzY3T46SmsnTCovmT5v8W8BWaIszGzicErPbLrq/ukI+C0z1oD39e/HKKYRUd9AeGCge6g7AkPJ7o0APRE9Fr5Msdl/a6Y7PEyosFXJCpH4kcGeVcZoIKwoYrA/0N/X29srW3dKb+ju7o5NePXJZ2sWXX4AECcl0pl9/iXfCT+3+tGenp5kZr96zX0TBII7gSPdnRv+dA8XNEoK9YwMjwS6urpkbza3cHQ6nZx3B4Z2vPjrL/AIVwKBQHj79u1mHDPL/OGWFgYROMNMsP/tdT8VQoQ55z1DQ0OBnp4eOdea84yNjY1s375dQ7DhsYUYQouixCKtrf4MeZiZMXMo/zD7habtGtyxZZeiKEPWgHWou7tbDjVmJrLVGX+7b+nviWi/mfZw8IIbVjQD4gQzXtfrz/+Vcz6s6/pAb2+vnBPMmUqi+v3+GAC+lgwAAFtx+Q8dDoec85JDfsSKVWYcCXMSuuyRw8PDsiJlfmTlyzwlfobk39bTFvL5fEPl08oH+vr68p4v09Mz+88LVUxGgAJzWCRwYIRrPBAMBod6enpk4000rqQ8Ut/o7u4OezyegJTnlFNOkdOWuR7iLJOV+eUbHp5EBPF5IpFYZKh3HVEsFI1GA/5ev0wog8EYri6E8XCCLh+XKewaMx6RiCIa0VgsNtzR0ZHeaMyocVgue1ICOa/mnBdVV1cntWEFWUrBSXxeUWVjERapqamRhZYyBKbwOxgnNm3aJCtbNqrxcNNIs3steqaCVek6yxljPNLd3Z2oxOzEJnnGlL0MeZLDrJWL75qXIHKxNeB/50UhLKPB4ECG5pSe4v6A/44TyxtuTQ/P4WdIeE5Ce5U4QottnVj3+WpeWlZQPb9yMrdYBGM8Q+A4rjAQQbcCJMUHhlhQMath+tZ3P+obUzwIdYymL3Smn7no19ue/9UXcsh1XIN1iIaVQ3plPK2SGep/9Wxr23AsEk72TAZicSpDYcSCPQOGERodG8ezFmyCRi6sZQNI+Mdz51+5wmJuOBKXWwvcE2rn/qm46qRNtgJHq8ItHzJkH2X7yTjkyr3padiKKiorKxXZM+P5Chr6/nSZGLfMnH3B5Rv7CwuLxxSUdDbHzb/h3uv3EIEcdZIfMlY5+/xvv++ct6AiIXcy8giBscpsYYB4aiotyuFFTsZSvc41vKaR0JupAdl9QuFJbTk7xicLZTZLsaZpCS0SNt1/7RCBeCudG2NK/Qz3/L6F1997+dEWYDrvw/tpXToOIquoOWPB3kVL7//JvHnzDg036YiH8ccJP3dtiVxTJnuppCGhH5CVWaQXHW4sTyZhGMZahStnJgNyAFaLOOeg1n4IITyw73fBvTte5cwaFow0ZPywDQiBOOikCDDiSk9o367tnKfSjegDF5RYJvYeSikJMW6xPti8bNUvtKg2/8X7r30jGXMcge5Bz7dOLG+QnST9Y8iVW4vP+LdlCz7ztcv+dvfVUtPPa6RLMIpXZqE1ZgWwJsLirhEOe6VmFXPEZA/Nj6nONwKHO1IYZfMgqemVCQg0tP3DrYqhDBmKEUqvlGxsZJgQAuU6UOKPjkIsGg1JTS+pBW+6p6Xv3KvvOM1hL3oTATNbPUKxpcDyetOylV0Q0y5ce9917+Wd31xCjRMup6PSS348tfCEyVsR0ZGOigAOReFPNi9f+ZCIxS5ce+/1r+QrT7w32rAkpVfKBEiPDeq6bgwNDR22hyQFYpitBySjkwBhWRIeA7jiKLeiNRjDWG8wGDwghOjJ5wcA+xFxn8QtL7cNSBU+PfMvP3BTa3B/p5uI+tPTTfgZ4lRms76zaNnKN8+94nY5fx2375Unbuse3bGjVgghbb45PixlVtvG5mUrvV9a8rPpOZBSguMtVY9GkNvSeqZhBBVFMaxWa96VSVwYaaN1SmIJDyEZ6RsWvMBep4EWHRkZCY2tZ/MbDRJMx3fFpsfv2Tpx4sQZp337upsVe8n1uQzuiHhGYXFJ3/yl9/3ni3dduzpvfWH89NNj6ZU193VXVVXVn/L1yy+ylFSsQMSk+TQFGdHpKK/c1bT0vofXvvfXq2DTppzLpHiPtBNmVBhX+ARpr0xhfBiPITC5xhsfFeXyIeVjinW2HC7HAo8o3RRGuT2ir69v9KM/r/r5/rfWzzGi4T/nRgWwcMtDTUsfeM7pdKa28vGIjiyOpJHA+8Kjv9/+whPVeiS0Ol3zNrNj3HJ50+kXvl5TMz+nPPHKHIpZ5byY+lkLkqay1IjcPm6BKbljD8UQgOeQ7yAklyqKovAJEyZkDPnpuEfhF3IYNka7u9pfWP29vg/ecZNh7M3Fj3G+aObCJdKWmmcjzcUpZ7iQVqhJkwr7Ojc8fkP3e+tmC8OQukrWjyGeXn/hAm9NTY3sxYmGn8SNF1xQVzKs+5YCx8lCCKbregZRkjoNUAQuSAvK6o2GjRRTXALJWjq5NBQKyaE/7zQTtEfgkjSPdXR0BA/seGvbx0/f4x7t2b2YhMgYLSRPxpULv/DvV7iPZw+VVia/3z8a3r9rj//RB84a3NU+D0h0ZcsTItbUnHfJYqfTmdHA4pUpj2IQUJrygkWGYbAqUXUEBUsXZBMgPaysfCirHbfSdebXFCW+6D+CNNO55+2njo6O6O7duwPtL/1x3fZXn3XrsVCK4T/BqXRyzUOhUEhqnsdz1CBpE94xuGN4z1svvNP614fmGqMjtydkMLtKcelPAoGANHok7dAyPimcoNTFrBz1OOfKSPFIchFuZpgNRkRXtvD0sDUtLVnXrpbi0mYAh83pdGYuIdKZHDu/HOrC/vc29bY+ec9lsWDw5nTWyJjT4XCU1NTUZPSGdNxj4KfOzs7I3i1bhj78y4rbwsH+S9LnUsb4nLKqWZOmTp0q589kw09WJujiTymCIPDy2Z+ttYateQ17zTf+UraUfD8iITLskcxirZtU55o2NDSUV5r5JpYnniFPE3Ss/829RMJvpkHGi62Fk8qHhoZSCs+McxxgQ87ve199+mkgkTEtlU4+carVak1RhpKVue6eazYQgDSoJ7/C6bMvJE55ZUBA0Z1JwjwAIYxV2dDKahpvRYyfi0m2uGx4xylMDr0xwzB+k87fUmAvdTgc/+pGJodeTYtFb0qXBy2siDEm6yZZh0lALrTJSO2dFntxs1bMHHKzNp2Z2S+PRTJGKZva5vhs8Lq7r10HBIH0OF5QcF7NaefXVldX59WI0umPgZ8QFGnKNH8iGjgQsVrzG6XMhMcAppgW7U7nExsOxnTdmjIFmisT1t511ffNRPJA0Ymf+cr5+/fvl/uCOXvKCeWaPJ1WaqbNBzaEuC4bXkntKX+vmDZHWolS5MuGezzCGMcvmvmS0HspFtPl0ZJ065IZ73jBjsKSRWbeJCg60tPZh5gqT3phkdBjzWaBiyZNf3jGjFMnNDY2ZlVKPn/Dw5MYQFatyyxANnjdXVf9lggyWh0ilkz5fNNHrs+dLxtIuozZWGUNa77xwa83LV2VthuUFTUZ2Ni4xIJIKduBsVDwDSLSYrG4nTqJe6RA07IHz5q/fMWXx+sYmTxbGCKmDLNGdLSVSNNsMT3Fbp5RUGvvvm4tkZGiok/5UvMHZc5za9JVYXkAeKJi7MsUIP+QWF+fG4iG0ikQcdLMc5p65l9zt2xcGXKm42fzExk2xqG1efnKPU3LV33tcHzmtTxWMOU89wcAqScC9m76y0od9UgsFpO7HRnWsmxpZwsToDELsPXNy1YNNN34wH/A4sXjTl+y0puXV6xHxBRjzIEPXr2TMRbpi/ZJeZLWsmy9jV745dWXLFr6QBVyHj8CKffbHCdO8y5atnK/IPEcR5xIyM5DgKTBXBj6NsaV2dkyMV7Y+t+0DH5h8ZW1ZbPmpM9TIJdHloKCZ5uXrQwYQr/t487dD3alnWJfcN2qWm6F20nQtLV3Xnl69rRwKgN4tnn5qjABbEOD/uv5u65ck8BtumHV51CBOzE8+jnA1GOj/W1vXwagjzDGhseOdiTIPrmLMIEh/23zrLN/BcvP6TQM8ct1d131eKKhLLzhYZUpxh0IIHtxisYa3O2/JdS/Z38spgxPmjQp2tfXN25lSiGN7etWL6g+7zs/5AWFd8pDzvJYLiJM4cj/UyKYJ1B5OHrPxidunrHwB+9/ghzSP9es7Kuqqir6zMU3b+AKl8f0zexRHvzi3HLXqbNm3XXq8qxKMCBie2Njo6W1tTVl6EmTx44ADcDxmeZsfEypEpA+vGfbLQM7W72apg0Gg8Hg2GGrNJZH40VZUbWcs980L19l0qCTO3iHmBMYo71d9/d+8NrLhDQwMNA1PDAwINfryS/n8CVV4s6Xfver7jdfmBELDt5IQqRZiOQ2me4d3u75Zuf6x5YZhpFyHCKZQn5A3Oj84R9/ed7Ivl1fEYa2OT+yQ1hCGNHR0dESs5lLCBwwD0OHsMeH9NDIhr3/eLap772XN7IY6xP9om/sEPT4hGOxOupiTFlKxRc0DHRorzU1MrdPxKLv7f/gpcX73n7+GR30vtHR0b5zzz1X7tsme6WkNrXFnMx4ZWWlvbCwsKioqKgwFotZpEFc2m3lrophGLqu6yFr1Dpae/GPUuyb0dHAX3e99PsbC4cLu1q7WzP2GbOkKOWxTp06VSZWMuWzC85zTJxyKVgstYioIEj1jQgJhDzKr0fDbwf2eH8zsrltV1RENR7lAXDAwMGjmEnuitPpnDjtnG9cabEXfwkVpQYwvkktbftyY1sWiCb0qC/U1fl4j/dVrxBctk15KHl4cHAwMHbsM+dcKUeEvr6+icUFxZUGM+TUFezt7d3f19cnD2WnFLjUO2pqTimf8cWmyxRH0XzFaq+XeSMgZpJHF7HYjshA9x/3vf/315nBhDyETkTD8qB0R0dHygZ8Iqf5VKbElT2YN1Q1WNkJzBKJRLhdt7OwEhaFsUK9X/RHOzs79eblq1K6fWy4/6nOV55qOYLKTMjFpamKMVZQYa+wh8Cw2Wxg1XVULNKgpgHoqBPXuSEsQl4v0KLRaGR4eDiUpeCxurraVlBQUKgoirSvSr6KYRhcUZT4KQXUkQxmCM65EY1GY1K5iEQiYel2dHRkNT0mBB1zWVVVlb24eIqjoECTjVwaHkbTGlWCBGtqaqw2m62QR7mD7GRlGlN0zhSFBJJCKOXRkRHnhhGNQqyIWSL94X7ZGSKdnZ05lbBsClAiUbMrW6Vo62nToScejIsXL8Y1a9bIVpdoeQk3SadFQvtk7/0E6q4xdkA5MnPezGCwq4vL0xSx2CiPRErjxxSCEISCggLDPmAXAUfAOPXUU/U1a9ZIOdPlkLbO6OLFi7UdO3YM9/f385KSEq7rOguHC5jU4ORNHcZGyeFwxIeZqVOnGpvG2QROZvAQIKQp8Oyzz4709vbKawxShlw9OW5lkk2ysbExmJAnGgrysBBYBmVj8gRIHgwwjJhWfMJUo3VTa2IiTc9fUop8e2aSIBfQuKTFcWL5RNn9k9/wTt+1Xa3/fEHXR/bmuhGVRM4PyCZvzswdhuWx5HWYpPKKPmp58u2Zh5Vm8oSJ8h5myhca2L/XYjF0AEUW+CctdDPPY8Ejwe9Y8krwPBr3qOXJqc0eqVTIQJ5BTX4kjODwno6ukRGhlZaW5hpykvifAkdfAsekZy6SdyABas3i6JGRtxH1qN1OsfR7hGa8T+FjVwLJnjnv+rsnLrpp1cZPwho5ZuxN7nvz+bsURQkNDGD+J+I/SeKf0iRLINkz7VarPIt3bvPyVUQkVmtR4xd/v//azBuySVKARTeuOAMQXwdINYFFhnr/GB0dDmqaNtLX1yEr86jnA1Oyn4I5SiBZmeZ4RPYDi40taV62qlMAvYUA6wyh+8EQIWSW6UzBBUgg71fOMtNJWA+HWve98ewjhmFI47m8+Zyy9kzH/9R/7EogOcyms4zf0kKYwRC/jYhPyptXitXWzhX2EgJem60ijUjog50bHrtO07SBQMAY6OjoONpLqjhnzpzDHkcZw5ENE2fOnJn3vmpNTY18ACJnGSTKZAwv4T1i92jp803wsBnJh5Ew9N2BDs9V2//+xJWI2G+1WnuKi1GeIkgsdPNhk4EzY4Y6yaJYt2ZEpAVYLbYOl6vhq06nc5rDXnjQrJGGk81rL3AE3G53iuKWAy9jiy4bnjlMVdWEds9kOtXV1fFb4qqqLjTjZYPdbneZWq+ely1uvLDkMPviHVf1nnX5HeWFCi6yFNh+xLjls+MRCmEMRId6H+v3v7VRH+gd1EGPGoY2zEesQ3qJPjJmBvuXzJUeb1v8urvT6czrTkYiXx5vW7aFeiL6E7tz5syZgoA/BAB5m1yY00FgTwNAxhM15sSQ8HbiuB0AXjKHHw7OyEx1dXUB57zEbrfL4c0KGlg451xeHZA/aZ6LG9iZIV/hiNtFNU0L22y2kHzIoaurS9oOj6pHJoSeMUOtKnTgR16f54REmFqnNqGCPxYEcv/zfZ/P83PV5e4mEBcR0XaGfBsCPE8AZ5IQj/o2+25xqw0fC4P+yjgsAMC5mh47ZcuWLR632kCowUxDIbnN50JCuwACn88zI5GedN1qg6wQ5nQ6T2Oo/AKIOpHhNw2hL/P7/StdLvdzCFgBRJVev6dBdbrfRAZ1ANDm8Xo+71YbtJgWLbcolscR2QUA9K4h6CmO+FVC/MDrbVuqqidfDyDmki7+gJz9DgAjiNQ9ODR4btmEsndJ4Hpk0OTxtsnjrFnLN2OYlWa3adOmDQwNDXXLW0oj0ZHdOuq7IkZkd1gP7w5poT1hPbxneHh4TzQa7QoEAnutVusB+ZiDPH+aKyFz4RwVzPEZQ8BqAHwWAX6ShZfi8Xku8vo8JzHOb3Y6nfFegBy+4fF6ziCg33Fu+baZzufzLPZ6PU4R30DBlNe2zHgSRoQzRsN0kxHT5zHkcVwEOF+QPt/r99TLO60xPSoPg2+XFWmm9/q8F0pjuQz3+70pT9ok8HxbfOsR4TlAWCXxiosnLCHCvYTgESS63U53yjmtBJ10k8OsKZDGHmVI3D+RvTfRg6WbGDqlG4e7uzOO8ZjYHVsQEa1+X5vclZdfylt90WiUHPZCKbccHeTHETE+9JKID1k6CNotX78ai0869fXq1YzBiMfb9pdkYBaABHXs3OntUVW1HGjsFADRnxnyYZfL9V2fz/dEFrLDBpEhSlMXeAdJGJJTkPiH3++VeU3JbzrTjJ6ZjjBWYfFdk7Fel4ATlZqF5LgGxeRwly2FcDgs718mGl4cJRrlcnkkVdacF4SmT59exhneZgijMRvflDDMvMHm9XsvBoPOQmCP19XVHdndThLxU/KM449S0hnzIEMfZyxnbzTT5FOZZvz/DbhYVdUHx37/TkLcwpBvrKmpmQoAKY8Idnd3h4hIqa1119XXu8+Wc2hHR1vWCzjmjJSUlG4HpOcQ8fsulyuvgjPTu13uVTroUpfQt2zZMrx169ZuIqqeOHFixgggN5jnzJlz5pQpUxzy5UwA/Jaquv9AJB5K8CRB20DQ+VID9nq98o7oZJfLHX+Qo76+Pqk/JPAT7v/Tlblzp7fXEPophmHcL3/hMNvg9XvvJhDVVqu1cdasWRNlRnRD+3wgEHhbwl6fh1ksYgqiIbw+jzxRCKGwaAqGgr+S8PDI8Apdj8Wv6uuGVtvW3rYHgKTm/jNEXG8YRoZJU5CRWL58ZAjtong6Xm+HPqodfFbUgMc48VKPt01etZNGEtJ0Nndy5eSzJa4go769vV2eOpDyTVcUpby0tLTU6/ferht08chI8EcjoyN3h8Oh6+M4fu89miFaCgsL5THRqMfbVkhkrHO73eds3rz5E2wPS66ffp+WwKcl8L9TAinKQj4iuKe7y6AY/i8AlIcioeXbt28f55GF8Tm6XO5bfD6P5HVEn8vlvhEPvls0aBjavZs3b96WDwN1tloPFviq1+/N+iKKy+X6N0RkXq/3GTM/l8t1JiK7FAgtBMbbPp8vOb+Z8Cwul/snPp/nZ6awfymYbWkyrgBURH8QJB63++3PecBzRJvOs2bNmmQvKPR5fW2VMhFEuBog3jDGTdMcqTrdrQDkHRjov7SoqMjBdFZujh8PJishEs/+EERc42UL6eByzVyZNoZsQ2A4MNVut+tE/OQcaUjtPifvHDTHNPiIKtPlctUQybfTeSyqRqeDF7bX19efhIjTwmG2Ta6/ZJm4a921nq2ebXPmNExvb2/bmZDYbrV/FhFK3bXuuvDWcDK8rs7tlvq5Z4sn/taBVO8ZY/Vc492ebZ4dCfq4i3ASEHwkNVep2wBA/Hin1A7LS8rnCiYCfr9fPnstP+50Ohulxcrn870XCAR2lpaW/mEsDtx17kYNwLpli0e+/ZNYVyei466qqm4iMGRFtre3y4PQr48hyKdM53LBCzxbPO9Kel2PJd+jV1W1nojKBwcHP5Syqqo6i4gGiaiIiKZs3rw5rrBJK1tdXV0j51y+ovmx5F1bW1tnsVgqOOcftbW1xc9VOZ3O05nBDG+7N+dB8yPSZomoAg8+ul8mBXW5XIsYU940DGNHkYO9XVdXVzt37twSsOJm1el+2aKIH6eUDMavM6ChYGVoSih+z0KNL9bpfFCwTZ2hVsmfolg+lssKsOFL9fX1cY0wwUfTYxcgY5eqLvVd845KeVlFF+jQw5F/T3Wq8buibrVhGyKWIbKLVZf667KysrkIPK6tqq6GdwSHxYhUrrrcKW/LJ9KSrtfrfR8BgxbF2uV2uS9LxKmqewNj7AzBsVR1uuXDS2C12OIV5HK670HCS4QQXeVlFXHDPxLbiID3I7IlCre85qxzflfSSFMkFzwEwFa66l03qqr6Y6ul4Coi2kPi4JVHt+reBDovJ4XJ6x85O+ARVabf738HAcOM0QeypTNgS1HQde3t7d2CaK3CLf+dyGxv/4GFXp83dc3GoFWq7X5/2z/HepY0Mlr9fu/dko4c5IRCvB0ByxVuka20mDP+9QRP6ba3t78xGhqZQARlVottuL5eXeh0un8g/+OBt8O7PRwN34SMyeFbfpN9Pt96XdduBYSUJYfX13a6z+dZvnmz92+AMO7LKh5f24kE4iFAfLhBbXjn4AyBX6Ihetrna/s7ofj9WHpxBxGuIMDvK9zyHiKG1Ro1vu9LADN8Pu9PhCEeRc7dqqpOQ8Cob6vvY9TpRwKEF4HJ3ZbFnCmyBw7U19erQDCRWXBpNBp6NtcIIhPOWctm4TeiFHUAAAKASURBVHLCCGUCRdyWh0B9AJhcJPf09KQcu8zFo6+vJ+WdWiSQxu63fD5P8t32dNodO3bI7bXZqupewRk+jABPETDziylW+ToIEck5jG/ZskVahp52u90H14WyR0iDPcdrBIDseYdVBH0+300ul+suhrx/zpw51VImvVCXe6cDPp/v0XQZBelf9vv8HybC3S75xPrB4zXIICYnWMMwShV+8Mk1b7u3TRrmVZebC8Imv69NNpr4V11d/ZmiouIX7QWOwalTp1aM2cAT0Un3iHpmkmoMIKJ2RD5PehHxFCKSc8e4HxGlXGPr7e2NL6aTRAxWI4A012Ut4NmzZ89M4BoGyjd6BoQunicS8RtpgUDAAkTd8jS51ExV9eT5CfwUl+Oz4Wj4W5oWjY8KKXGpnuQtLJ/Pl7i7IhuOoShKyj1OE1kXAs/QahlASl43b97sBcBSl8t1SKlC2sqRHjDxAvlghdfr+T9E8OcJJRO+Y44zw0fcM4loQN6QkkyGAkNLJpSWvetyufoJ8CSf33OK3FW3FzgyLhlJfNbLdouJtFd1NTzl9bVdRHRQeRkTSNLEPB7PG26X+y+qy70XkJ7UNHqnvd2XfE3LZrU/obrcspnHAOjAaEifv3Pn5l2qy92pOtWbAfEig4xvSp5CM76ICntKVdW1QoC8jLQJSI4g8e8Nm9X+LhA8nZBDmtoo7TzTzJkz7YWOoi4CsgCBEDq1yGll9mx3bYGVv6y6Gk4jEFvk8EkE8XxLy5Oquj+WW2EA+Ob+A923ynf7cOzNCJKVSvJpurh56HuI7BWXy/0UIn3g9XrPcavu1jGt/ZWR0MhtRUXFLyNAjOJbbN5LD4qf+fd/ACIieKbyCmQ9AAAAAElFTkSuQmCC"
    }
   },
   "cell_type": "markdown",
   "metadata": {
    "kernel": "SoS"
   },
   "source": [
    "![image.png](attachment:image.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "SoS"
   },
   "source": [
    "## GNU Parallel"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "SoS"
   },
   "source": [
    "[GNU parallel](https://www.gnu.org/software/parallel/) is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU parallel can then split the input and pipe it into commands in parallel.\n",
    "\n",
    "When you hit:\n",
    "```\n",
    "parallel --citation\n",
    "```\n",
    "The notification goes as:\n",
    "\n",
    "> Academic tradition requires you to cite works you base your article on.\n",
    "When using programs that use GNU Parallel to process data for publication\n",
    "please cite:\n",
    "\n",
    ">@article{Tange2011a,\n",
    "  title = {GNU Parallel - The Command-Line Power Tool},\n",
    "  author = {O. Tange},\n",
    "  address = {Frederiksberg, Denmark},\n",
    "  journal = {;login: The USENIX Magazine},\n",
    "  month = {Feb},\n",
    "  number = {1},\n",
    "  volume = {36},\n",
    "  url = {http://www.gnu.org/s/parallel},\n",
    "  year = {2011},\n",
    "  pages = {42-47},\n",
    "  doi = {http://dx.doi.org/10.5281/zenodo.16303}\n",
    "}\n",
    "\n",
    ">(Feel free to use \\nocite{Tange2011a})\n",
    "\n",
    ">This helps funding further development; AND IT WON'T COST YOU A CENT.\n",
    "If you pay 10000 EUR you should feel free to use GNU Parallel without citing.\n",
    "\n",
    ">If you send a copy of your published article to tange@gnu.org, it will be\n",
    "mentioned in the release notes of next version of GNU Parallel.\n",
    "\n",
    "**So don't forget to cite [Tange2011a](http://dx.doi.org/10.5281/zenodo.16303) when you use GNU Parallel in an academic study**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "SoS"
   },
   "source": [
    "## First steps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "SoS"
   },
   "source": [
    "### Jupyter and markdown basics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "SoS"
   },
   "source": [
    "First of all, please watch this short video on how to use the modes and shortcuts in Jupyter notebooks:\n",
    "\n",
    "[![The Data Incubator - Keyboard Shortcuts in Jupyter](https://img.youtube.com/vi/cuHY1o3Cf2s/0.jpg)](https://www.youtube.com/watch?v=cuHY1o3Cf2s&index=4&list=PLjDTd-bDo6Q3nnt7y_GjMaYD79-stYZ-O&t=0s)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "SoS"
   },
   "source": [
    "And you may also want to review below links to get an undertanding of markdown syntax, to create a better layout and view of your homeworks and/or your own Jupyter notebooks:\n",
    "\n",
    "[Markdown Cheatsheet](https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet)\n",
    "\n",
    "[Markdown Quick Reference](https://en.support.wordpress.com/markdown-quick-reference/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "SoS"
   },
   "source": [
    "### Journey to Linux: Breaking Windows habits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_style": "split",
    "kernel": "SoS"
   },
   "source": [
    "<!--In ![](https://thgtr.com/wp-content/uploads/2009/10/windows_logo.jpg.png =10x) -->\n",
    "\n",
    "<img src=\"https://thgtr.com/wp-content/uploads/2009/10/windows_logo.jpg.png\" width=\"100\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_style": "split",
    "kernel": "SoS"
   },
   "source": [
    "<img src=\"https://openclipart.org/download/250889/Pinguino-Linux.svg\" width=\"100\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_style": "split",
    "kernel": "SoS"
   },
   "source": [
    "We have letters for drives/volumes and each drive has its own root as such: C:\\, D:\\ .. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_style": "split",
    "kernel": "SoS"
   },
   "source": [
    "We have a single filesystem root \"/\" and all drives/volumes are \"mounted\" under this root"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_style": "split",
    "kernel": "SoS"
   },
   "source": [
    "Directories are separated by backslashes \"\\\\\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_style": "split",
    "kernel": "SoS"
   },
   "source": [
    "Directories are separated by slashes \"/\". Backslash is an escape character to toggle between literal and special usages of some characters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_style": "split",
    "kernel": "SoS"
   },
   "source": [
    "Directory and filenames are case insensitive: \"path\" and \"PATH\" are the same"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_style": "split",
    "kernel": "SoS"
   },
   "source": [
    "Directory and filenames are case sensitive: \"path\" and \"PATH\" are different and can co-exist"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_style": "split",
    "kernel": "SoS"
   },
   "source": [
    "In Windows tradition, directory and filenames contain many spaces since it does not matter in GUI usage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_style": "split",
    "kernel": "SoS"
   },
   "source": [
    "In Linux tradition, whitespaces are avoided as much as possible because they may cause trouble for shell commands"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_style": "split",
    "kernel": "SoS"
   },
   "source": [
    "GUI's are the main interface for interaction with the OS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_style": "split",
    "kernel": "SoS"
   },
   "source": [
    "[A true Linux/Unix \"hacker\"](http://www.catb.org/jargon/html/appendixc.html) prefers the terminal/CLI/shell (the black screen) for many good reasons!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "SoS"
   },
   "source": [
    "### The Shell"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "SoS"
   },
   "source": [
    "<!--<img src=\"https://static01.nyt.com/images/2018/05/15/arts/01hal-voice1/merlin_135847308_098289a6-90ee-461b-88e2-20920469f96a-superJumbo.jpg?quality=90&auto=webp\" width=\"300\">-->\n",
    "\n",
    "- The shell is the command interpreter of Unix/Linux systems\n",
    "- The terminal is the windows program that a shell works on in desktop environments\n",
    "- When we work in a virtual terminal, no need for a separate terminal program, everything IS inside the terminal. Try it yourself if you are on a \\*nix PC: Ctrl+Alt+F2 and login! And back here with Ctrl+Alt+F1\n",
    "- Together we call this mode of interaction with the OS \"Command Line Interface\" or CLI"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "SoS"
   },
   "source": [
    "In Stanley Kubrick's sci-fi classic 2001: A Space Odyssey, an AI interface to the main computer of the spaceship looked and talked like that:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "SoS"
   },
   "source": [
    "[![](https://img.youtube.com/vi/7qnd-hdmgfk/0.jpg)](https://www.youtube.com/watch?v=7qnd-hdmgfk)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "SoS"
   },
   "source": [
    "Hopefully the shell we use, BASH, do not refuse our requests arbitrarily, as HAL does to Dave"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "SoS"
   },
   "source": [
    "In an SoS notebook, in order to invoke Bash (send a command to be interpreted with Bash), type the command to a cell with \"Bash\"at the right and hit Enter: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "# Hello the commands you type and enter here are interpreted by Bash\n",
    "# But statements that start with a hash sign \"#\" are comments not interpreted at all\n",
    "# So that's a comment that tells about comments!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "We can also invoke Bash through a terminal.\n",
    "\n",
    "In \\*ubuntu distros, Ctrl+Alt+T is the default shortcut to invoke a terminal window. If you are on an \\*ubuntu system, try it yourself, and to close it type \"exit\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "Now let's open a terminal from inside Jupyter/binder:\n",
    "\n",
    "(what exactly the following code does is not important for the time being)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "last=$(( $(ps aux | awk -F \" \" '$7 ~ /pts/ { print $7 }' | grep -Po \"\\d+\" | sort -nu | tail -1) + 1 ))\n",
    "echo $last"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "echo \"Follow the working one of the following links to open a terminal: (that matches the domain of the URL above)\"\n",
    "echo \"https://hub.gke.mybinder.org/user/serhatcevikel-bdm_2019-$(hostname | grep -Po '(?<=2019-2d).*?$')/terminals/$last\"\n",
    "echo \"https://hub-binder.mybinder.ovh/user/serhatcevikel-bdm_2019-$(hostname | grep -Po '(?<=2019-2d).*?$')/terminals/$last\"\n",
    "echo \"https://notebooks.gesis.org/binder/jupyter/user/serhatcevikel-bdm_2019-$(hostname | grep -Po '(?<=2019-2d).*?$')/terminals/$last\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "Now please copy and paste the below command inside that terminal:\n",
    "\n",
    "```Bash\n",
    "screen -S 1\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "A very powerful utility called \"screen\" is automatically opened inside the terminal"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "From now on, we can send commands to the terminal from this Jupyter notebook using the utility called \"GNU Screen\".\n",
    "\n",
    "We will come to screen later, however, what it does are:\n",
    "- To multiplex many shell instances inside a session\n",
    "- To attach and detach to an ongoing process, from local and from remote hosts (think that you were disconnected from a remote computer, you want to reconnect and resume from where you were last. screen does that)\n",
    "- To send commands to other shell instances easily\n",
    "\n",
    "Note that, it may take some time for the screen inside terminal to be operational. We will go on and return back to screen terminal in order to send and execute some of our codes to terminal ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "And this code is getting the appropriate parameter in order to send remote commands to terminal from here:\n",
    "\n",
    "(content not important for the time being)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "#scr=$(screen -ls 2> /dev/null | grep -Po \"^.+(?=\\.jupyter)\" | head -1 | tr -d \"\\t\")\n",
    "scr=$(screen -ls | grep -P \"Attached\" | head -1 | grep -Po \"^\\t+.+?(?=\\s|\\t)\" | tr -d \"\\t\")\n",
    "echo $scr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "### Get info on commands"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "SoS"
   },
   "source": [
    "We do not usually memorize all usage details of all commands. We can access this info whenever we want:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "To get a one-liner explanation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "whatis ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "SoS"
   },
   "source": [
    "To get most verbose info:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "man ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "And best of both worlds, info on most common usages:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "tldr ls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "tldr tldr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "Note that, tldr is not a standard utility in \\*nix system and you have to install it as an npm package separately. It is a community project and it may not cover many lesser known utilities/programs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "And for screen:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "whatis screen\n",
    "tldr screen"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "### ls command"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "whatis ls\n",
    "tldr ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "We see that, most common flags are:\n",
    "- a: all incl. hidden\n",
    "- l: long and detail\n",
    "- h: human readable units\n",
    "- S: size sorted\n",
    "- t: time sorted\n",
    "- r: reverse sorted\n",
    "- 1: one file per line (default in l)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "Let's list the contents of the root directory, with details and time sorted:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "ls -lt /"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "**EXERCISE 1:**\n",
    "\n",
    "Now try yourself: List all the contents of the \"usr\" directory right under root, including hidden files, and reverse size sorted, one file per line and without details:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "**SOLUTION 1:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "pass1=\n",
    "encrypt=\"U2FsdGVkX1+dE49K4/94vw8AdJTGsjV5Bza7+BMfh70=\"\n",
    "solution=$(echo $encrypt | openssl enc -md sha256 -aes-128-cbc -a -d -salt -pass pass:$pass1 2> /dev/null)\n",
    "echo \"$solution\"; echo\n",
    "for l in \"$solution\"; do eval \"${l}\"; done"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "### echo command"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "whatis echo\n",
    "tldr echo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "### \">\" redirect"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "\">\" redirect operator \"redirects\" the standard input (stdin) from a command into a file (and saves)\n",
    "\">>\" appends, \">\" overwrites"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "echo \"Hello World\" > ~/helloworld\n",
    "echo \"Hello World a second time\" > ~/helloworld # this will overwrite the file\n",
    "echo \"Hello World a third time\" >> ~/helloworld # this will append the file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "ls ~/helloworld"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "We see the file is there, but how can see the contents?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "### cat command"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "Cat \"concatenates\" or combines and prints the contents of the files provided as arguments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "whatis cat\n",
    "tldr cat"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "Let's view the main file for configuration, environment and options of the Bash shell for the current user:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "cat ~/.bashrc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "**EXERCISE 2:**\n",
    "\n",
    "View the contents of the helloworld file at the home directory created recently"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "**SOLUTION 2:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "pass1=\n",
    "encrypt=\"U2FsdGVkX1/+/ZVgeVyUxQ9lO6wDAKaPxNKIkrz8LigmUJPhOkJxvHhiw0/Zi7xw\"\n",
    "solution=$(echo $encrypt | openssl enc -md sha256 -aes-128-cbc -a -d -salt -pass pass:$pass1 2> /dev/null)\n",
    "echo \"$solution\"; echo\n",
    "for l in \"$solution\"; do eval \"${l}\"; done"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "### wc command"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "Most of the time, we may be in a need to get the row/word/character/byte count of the output of a previous command"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "whatis wc\n",
    "tldr wc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "See how many files/directory there are instantly under the root directory:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "ls / | wc -l"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "We are ok with \"ls /\" and \"wc -l\" parts, but what about that strange \"|\" sign?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "### Unix pipe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "Unix pipe \"|\" is a structure in Unix-like OS's, that redirects the standard output (stdout) of a former command into the standard input (stdin) of a latter command"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "ls / | cat -n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "The first process writes the contents of the root /, and the output is fed into the input of \"cat\" with \"n\" flag for line numbers\n",
    "\n",
    "So the command takes the raw input and adds line numbers. See how many files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "ls / | wc -l"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "First process does the same thing, second counts the lines in the input"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "Here Prof. Brian Kernighan - one of the pioneers of the Unix systems and C programming language explains how the Unix pipe can be utilized so that simple tools are combined to solve more complex problems:\n",
    "\n",
    "[![AT&T Archives: The UNIX Operating System (Unix Pipes part)](https://img.youtube.com/vi/L9GfCgLLZYE/0.jpg)](https://www.youtube.com/watch?v=L9GfCgLLZYE&index=11&list=PLjDTd-bDo6Q0U5DfSP0tOjyRrG218anv7&t=0s)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "Pipe is very powerful and simplifies complex data and workflows!\n",
    "\n",
    "You'l create killer one-liners that does a lot of things using multiple pipes!!!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "SoS"
   },
   "source": [
    "### Environment variables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "SoS"
   },
   "source": [
    "In our workflow we may refer to same paths and filenames many times, and some values and parameters may be selected once and referred to in many places\n",
    "\n",
    "It is better that we define them once in an environment variable and refer by that variable name:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "First, some built-ins:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "echo $HOME # echoes the value of HOME env variable\n",
    "echo ~ # tilde is a shorthand for HOME\n",
    "echo $PATH # echoes the value of PATH env variable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "$PATH is where the OS looks for the executable files in return for a command"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "For example when we just type \"ls\", we do not specify the exact location of the \"ls\" executable. The exact pathof the executable file that ls command invokes is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "which ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "To list the details of that file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "ls -l `which ls`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "The backticks execute the command inside and provide the output as an argument. The same is done with:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "ls -l $(which ls)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "\\$(...) notation executes the command inside and saves it as a temporary variable to be used as arguments"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "Apart from /bin/ls, for all ls executables in whole $PATH locations:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "which -a ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "- Now let's define the path to data directory, just under the \\$HOME.\n",
    "- Note that we do not use the \"$\" when the variable is on the LHS to an assignment:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "datadir=~/data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "Now let's check the value (with \"$\"):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "echo $datadir\n",
    "ls $datadir"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "Now let's define the path to the directory for \"imdb\" database, just under datadir:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "imdbdir=$datadir/imdb\n",
    "echo $imdbdir\n",
    "ls $imdbdir"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "***IMPORTANT!: FOR THE SUBSEQUENT CODE SNIPPETS TO WORK WELL \\\\$datadir AND \\\\$imdbdir ENVIRONMENT VARIABLES SHOULD BE DECLARED AS ABOVE. IF FOR ANY REASON THE BASH KERNEL IS RESTARTED, THESE VARIABLES MUST BE DECLARED AGAIN BY RUNNING THIS CODE CELL:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "datadir=~/data\n",
    "imdbdir=$datadir/imdb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "**EXERCISE 3:**\n",
    "\n",
    "Define the \"imdbtsv\" variable for the path to the tsv directory under $imdbdir, by using $imdbdir, check the value and contents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "**SOLUTION 3:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "pass1=\n",
    "encrypt=\"U2FsdGVkX1+eMbt2jVy8BK8OBgzJXwsKuew6aNDkBvUziV/0dZm7RhCnl2YJCX1c YZ0alY4WMRt7ptJUPuimyde+oTl/mPvXJCkF5xXYePE=\"\n",
    "solution=$(echo $encrypt | openssl enc -md sha256 -aes-128-cbc -a -d -salt -pass pass:$pass1 2> /dev/null)\n",
    "echo \"$solution\"; echo\n",
    "for l in \"$solution\"; do eval \"${l}\"; done"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "## imdb dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "Let's view the contents of the tsv directory:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "ls -lh $imdbdir/tsv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "tsv files are \"tab separated values\" but here they are gzipped for portability concerns. We will inflate the files but first let's get info on the contents of this database:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "Contents:\n",
    "\n",
    "The database is separated into 6 tsv files:\n",
    "\n",
    "* title.basics.tsv: 365 MB, 4,534,355 lines, 9 fields. Contains the following information for titles:\n",
    "    * tconst (string) - alphanumeric unique identifier of the title\n",
    "    * titleType (string) – the type/format of the title (e.g. movie, short, tvseries, tvepisode, video, etc)\n",
    "    * primaryTitle (string) – the more popular title / the title used by the filmmakers on promotional materials at the point of release\n",
    "    * originalTitle (string) - original title, in the original language\n",
    "    * isAdult (boolean) - 0: non-adult title; 1: adult title.\n",
    "    * startYear (YYYY) – represents the release year of a title. In the case of TV Series, it is the series start year.\n",
    "    * endYear (YYYY) – TV Sereis end year. ‘\\N’ for all other title types\n",
    "    * runtimeMinutes – primary runtime of the title, in minutes\n",
    "    * genres (string array) – includes up to three genres associated with the title\n",
    "\n",
    "\n",
    "* title.crew.tsv: 133 MB, 4,534,355 lines,  3 fields. Contains the director and writer information for all the titles in IMDb. Fields include:\n",
    "    * tconst (string)\n",
    "    * directors (array of nconsts) - director(s) of the given title\n",
    "    * writers (array of nconsts) – writer(s) of the given title\n",
    "\n",
    "\n",
    "* title.episode.tsv: 72 MB, 2,986,926 lines, 4 fields. Contains the tv episode information. Fields include:\n",
    "    * tconst (string) - alphanumeric identifier of episode\n",
    "    * parentTconst (string) - alphanumeric identifier of the parent TV Series\n",
    "    * seasonNumber (integer) – season number the episode belongs to\n",
    "    * episodeNumber (integer) – episode number of the tconst in the TV series.\n",
    "\n",
    "\n",
    "* title.principals.tsv: 281 MB, 4,008,569 lines, 2 fields. Contains the principal cast for titles:\n",
    "    * tconst (string)\n",
    "    * principalCast (array of nconsts) – title’s top-billed cast\n",
    "    * Note: This file will be pre-processed so that the second field is “molten” into separate cast for each line (SÇ)\n",
    "\n",
    "\n",
    "* title.ratings.tsv: 13 MB, 767,042 lines, 3 fields. Contains the IMDb rating and votes information for titles\n",
    "    * tconst (string)\n",
    "    * averageRating – weighted average of all the individual user ratings\n",
    "    * numVotes - number of votes the title has received\n",
    "\n",
    "\n",
    "* name.basics.tsv: 474 MB, 8,155,448 lines, 6 fields. Contains the following information for names:\n",
    "    * nconst (string) - alphanumeric unique identifier of the name/person\n",
    "    * primaryName (string)– name by which the person is most often credited\n",
    "    * birthYear – in YYYY format\n",
    "    * deathYear – in YYYY format if applicable, else ‘\\N’\n",
    "    * primaryProfession (array of strings)– the top-3 professions of the person\n",
    "    * knownForTitles (array of tconsts) – titles the person is known for"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "Note that title.basics.tsv is 365MB while title.basics.tsv.gz is 79MB. That's why large files are gzipped. Now we will gunzip them"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "### view gzipped files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "We can view the contents of gzipped files:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "whatis cat\n",
    "tldr cat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "whatis zcat\n",
    "tldr zcat"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "\"cat\" command lets us print the contents of a text formatted file to terminal, zcat does that for gzipped files. However the files are big and doing so will only show the last part in a terminal, and here in jupyter it will enter into a long running process that will overuse the memory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "We will invoke a separete terminal and view the files with a pager called less:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "whatis less\n",
    "tldr less"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "whatis zless\n",
    "tldr zless"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "scr=$(screen -ls | grep -P \"Attached\" | head -1 | grep -Po \"^\\t+.+?(?=\\s|\\t)\" | tr -d \"\\t\")\n",
    "screen -S $scr -X stuff \"zless -N $imdbdir/tsv/name.basics.tsv.gz\\n\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "N flag shows line numbers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "- Get help on less usage by typing \"h\".\n",
    "- Scroll back and forth with pgup/down or j/k\n",
    "- search with \"/pattern\" and N/n gets to next/previous match.\n",
    "- xxg goes to xxth line.\n",
    "\n",
    "Now let's play a game in pairs:\n",
    "\n",
    "- Search for your favorite act(ress|or) and tell the line number to your partner. (S)he will find that act(ress|or) and tell you back!\n",
    "\n",
    "My ones are 794034 and 200. Who are they? Hoo-Hah!\n",
    "\n",
    "\"less is more\"! You'll love \"less\" in time. Hit \"q\" to exit ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "### gunzip tsv files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "First we will create a separate directory for gunzipped tsv files, just under $imdbdir. However if the command is rerun, it should not raise an error:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "whatis mkdir\n",
    "tldr mkdir"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "p flag to mkdir creates directories recursively and does nothing if they exist:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "ls $imdbdir\n",
    "mkdir -p $imdbdir/tsv2 && echo \"$imdbdir/tsv2 created\" ## second command executed only if first one is successful\n",
    "ls $imdbdir"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "Now let's gunzip tsv.gz files, keeping the original gz files and not overwriting any gunzipped files if the command is run more than once:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "yes n | gunzip -k $imdbdir/tsv/*.gz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "whatis yes\n",
    "tldr yes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "The first part repeatedly output an \"n\" for \"NO\" as long as the second part says \"should I overwrite the existing file\"."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "Now copy those gunzipped files into the new directory so that we don't mistakenly modify the original gz files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "whatis cp\n",
    "tldr cp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "cp -n $imdbdir/tsv/*.tsv $imdbdir/tsv2/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "Note that \"n\" flag is \"no-clobber\" which stands for \"do not overwrite existing files\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "Now, I am fed up with this tsv2 directory and get rid of it completely:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "whatis rm\n",
    "tldr rm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "We write a short shell function that checks whether the first argument ($1) exists as a condition and returns true or false. \"-e\" means \"exists\" "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "exists()  if [ -e $1 ]; then echo \"true\"; else echo \"false\"; fi\n",
    "exists $imdbdir/tsv2\n",
    "rm -r $imdbdir/tsv2 && echo \"$imdbdir/tsv2 deleted\"\n",
    "exists $imdbdir/tsv2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "Now rerun the above commands to recreate the tsv2 directory and copy tsv files\n",
    "Check the contents of the directory:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "mkdir -p $imdbdir/tsv2 && echo \"$imdbdir/tsv2 created\" ## second command executed only if first one is successful\n",
    "cp -n $imdbdir/tsv/*.tsv $imdbdir/tsv2/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "ls $imdbdir/tsv2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "**EXERCISE 4:**\n",
    "\n",
    "1. Write a function exists2 that echoes \"exists\" or \"notexists\" based on the exists condition\n",
    "2. Create a directory tsv3, but it should not complain if run twice. If executed it should return a message that it did so (even if it does not redo)\n",
    "3. Check whether tsv3 exists\n",
    "4. Copy the tsv files under tsv directory here, but it should not overwrite the files\n",
    "5. List the contents of tsv3 directory\n",
    "6. Remove the tsv3 directory and return a message that it did so\n",
    "7. and check whether tsv3 exists anymore using the above function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "**SOLUTION 4:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "pass1=\n",
    "encrypt=\"U2FsdGVkX1/vzhgzmriq2SUeRoCJn1hpgV0lP8qU9HIotS6WVfVmnnxMf5w5LgS2 uhJDLm+T+kuLmwDbYRF0CJc1kckMA468XJ4QVNyXzr8TWW+oIMylnPX15OXi3FFz d891bcFNirD8Q0bYiaCYuOrUi20JScwax1Ua9xPZtNOiIwoQHQbmUJ6EsaHlPkAM XdQJg7eZe9+9+YJCwcSswXda2fCs4/NIP2OVEXgHrhNZIgQXYAZ+K6R82EvGvGf2 vFia5CFXlunH20QWvJUkw2Ro7I7lMHa/U2c7UdQzkdyy28AJqFQhWV5J49Am2dOW pEWorE7btQ1gQ8PznqBFq9WfoQOviioxuIRD+XqLqSXO+PMbqmtW5TZfbDvYtDLL 3c71Fsb+AicJ9Fw5WxI+ig==\"\n",
    "solution=$(echo \"$encrypt\" | openssl enc -md sha256 -aes-128-cbc -a -d -salt -pass pass:$pass1 2> /dev/null)\n",
    "echo \"$solution\"; echo\n",
    "for l in \"$solution\"; do eval \"${l}\"; done"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "### Get first or last n lines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "Let's see the initial or last lines of a file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "whatis head\n",
    "tldr head"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "cat $imdbdir/tsv2/name.basics.tsv | head -10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "But the columns are not aligned"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "whatis column\n",
    "tldr column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "cat $imdbdir/tsv2/name.basics.tsv | head -10 | column -t"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "### Row counts and many ways of looping in Unix/Linux"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "Now I want to get a feel of how tall the tsv files are in terms of row count. For just one file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "cat $imdbdir/tsv2/name.basics.tsv | wc -l"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "This is hard to read as human beings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "whatis numfmt\n",
    "man numfmt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "cat $imdbdir/tsv2/name.basics.tsv | wc -l | numfmt --to=si"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "But single file does not suffice, let's do that for all tsv files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "ls $imdbdir/tsv2/*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "ls $imdbdir/tsv2/* | \\\n",
    "while read l;\n",
    "do\n",
    "    cat \"$l\" | wc -l | numfmt --to=si;\n",
    "done"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "That's better. We loop through the full path of the contents of tsv2 directory, assign each to variable \\\\$l and run the above pipe for each $l.\n",
    "\n",
    "Better, we can use a for loop for this iteration:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "for l in $imdbdir/tsv2/*; \\\n",
    "do\n",
    "    cat \"$l\" | wc -l | numfmt --to=si;\n",
    "done"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "But shouldn't it be better if we also report respective filenames?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "whatis printf\n",
    "tldr printf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "for l in $imdbdir/tsv2/*; \\\n",
    "do\n",
    "    printf \"%s\\t\" $l;\n",
    "    cat \"$l\" | wc -l | numfmt --to=si;\n",
    "done"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "We can use the more powerful \"find\" command instead of ls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "whatis find\n",
    "tldr find"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "find $imdbdir/tsv2 -mindepth 1 | \\\n",
    "while read l;\n",
    "do\n",
    "    printf \"%s\\t\" $l;\n",
    "    cat $l | wc -l | numfmt --to=si;\n",
    "done"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "But we can replace the whole loop with sth more terse (or concise) and aligned columns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "whatis xargs\n",
    "tldr xargs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "find $imdbdir/tsv2 -mindepth 1 | \\\n",
    "    xargs -i sh -c 'printf \"%s\\t\" {}; cat {} | wc -l | numfmt --to=si' | \\\n",
    "    column -t"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "Or do the loop inside the \"-exec\" argument of find command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "find $imdbdir/tsv2 -mindepth 1 \\\n",
    "    -exec sh -c 'printf \"%s\\t\" {}; cat {} | wc -l | numfmt --to=si' \\; | \\\n",
    "    column -t"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "Or instead of serial processing, process multiple lines at the same time utilizing multiple CPU cores:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "whatis parallel\n",
    "tldr parallel"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "And to support the development of GNU Parallel project, please read:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "parallel --citation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "find $imdbdir/tsv2 -mindepth 1 | \\\n",
    "    parallel -k -j0 'printf \"%s\\t\" {}; cat {} | wc -l | numfmt --to=si' | \\\n",
    "    column -t"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "j flag provides number of parallel threads, j0 means maximum threads available, k keeps the original order of the inputs\n",
    "\n",
    "Contrary to sequential single-threaded processing, in parallel processing, without k flag, the order of the output may be different based on completion times of each thread"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "You see there are many ways to do the same thing in Linux"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "**EXERCISE 5:**\n",
    "\n",
    "Find the WORD counts (not line) of all gzipped files under tsv directory, using any of the methods you like above. Check the \"name\" argument to file, the glob for the files is \"*.gz\"\n",
    "\n",
    "IMHO, this is a \"big data management\" course, and in order to process larger datasets easily utilizing the max power of your processor with full throttle, \"parallel\" should be your way!\n",
    "\n",
    "Note that in order to use text processing commands (cat, less etc.) on zipped files, we add the prefix \"z\" to those commands "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "**SOLUTION 5:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "pass1=\n",
    "encrypt=\"U2FsdGVkX18fvmKge1qRd1u+3HhdU41PB1axe167aRS527dX9nIx2ckFe3wPmE0E vQ93TpCe1jTphNkfZ+qyzB1oiJvxdVUclalfFlQbzhy42p1pN46/DMHvLGBho6RI oXelzh1SLtqk2O9YU4v3mwFcf5LtTZiSyq237Lni/Y5/3ZB0nePZz6TCt3i6Lavz Id7SgTK7XNsdEjtjsESvig==\"\n",
    "solution=$(echo $encrypt | openssl enc -md sha256 -aes-128-cbc -a -d -salt -pass pass:$pass1 2> /dev/null)\n",
    "echo \"$solution\"; echo\n",
    "for l in \"$solution\"; do eval \"${l}\"; done"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "### Column counts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "What if we want to get the column counts of each file?\n",
    "\n",
    "Combining all we have learned above, we can read the first line and count the words!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "find $imdbdir/tsv2 -mindepth 1 | \\\n",
    "    parallel -k -j0 'printf \"%s\\t\" {}; head -1 {} | wc -w' | \\\n",
    "    column -t"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "**EXERCISE 6:**\n",
    "\n",
    "gzip algorithm can implement compressions up to two orders of magnitude.\n",
    "\n",
    "In some cases it is not convenient to gunzip the files since they may fill up the drive.\n",
    "\n",
    "So we may have to do the same thing on the gzipped files, off you go!\n",
    "\n",
    "Note: There is no \"z\" version for head, but \"head\" can work on stdin "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "**SOLUTION 6:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "pass1=\n",
    "encrypt=\"U2FsdGVkX19GPSAW6SDyYhMan0hv+ZxMF544aTgc3uqtBfkllke344tehTZK1lBo I9x/LzmrmFnoY+SWtvstpln4pkqKMycuFxVuV1oCwEAn0FKHbUOqqbKthawGVWJb yJYo7jztKKIO/5/VROag1QHWQ07ihGUeNRMkGrr9CJQge2+pQyR2W4DOl4oKrwA9 /fuZ2du4+1tmXWM/Q48RIA==\"\n",
    "solution=$(echo $encrypt | openssl enc -md sha256 -aes-128-cbc -a -d -salt -pass pass:$pass1 2> /dev/null)\n",
    "echo \"$solution\"; echo\n",
    "for l in \"$solution\"; do eval \"${l}\"; done"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "### Head and tail of files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "Statistics are good to have but it would be better to review the structure of the files, for example the headers and first data rows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "find $imdbdir/tsv2 -mindepth 1 | \\\n",
    "    parallel -k -j0 'printf \"%s\\n\" {}; head -2 {} | column -t; echo'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "**EXERCISE 7:**\n",
    "\n",
    "Did you realize that sth was different here as compared to previous ones? What is that?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "**SOLUTION 7:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "pass1=\n",
    "encrypt=\"U2FsdGVkX1+2dKu4xqou48SPsqsiI6GE3dC+Otn8k6ZOHn5yUo21hVBLeZJmm6VA 2yWJ7AD4JwQ6Onk1oF6mT69Lno0m1k3tGRqUfB2A/YVf4H1CxvnC/1hGnMxBOBHM ZV72KSZKzb2X3xUY9MX0Yj1Pr+gYmEzoOB1NntJ7SfI=\"\n",
    "solution=$(echo $encrypt | openssl enc -md sha256 -aes-128-cbc -a -d -salt -pass pass:$pass1 2> /dev/null)\n",
    "echo \"$solution\"; echo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "If we want to get the last n lines, we use tail:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "whatis tail\n",
    "tldr tail"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "**EXERCISE 8:**\n",
    "\n",
    "We want to print the first and last lines of each file.\n",
    "\n",
    "Note that, in order the align both lines, they must be fed into the pipe before \"column\" command together. Can you figure out a way for this?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "**SOLUTION 8:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "pass1=\n",
    "encrypt=\"U2FsdGVkX18QvsiUoHDi8Xk/h3bH6K2gaaPXrVnxqxQ1eXDPTss1htcSDx5uOVW/ QNXlXUvh94Jk/yMPS0P+4LIKMhsnMPTDTAb7YdYF+DZ4jP285gQqBdezPyEg8qCc tNuDVOvwjjchyYm0pSoeDJHGAAgxrvapOQj3R2hi2L8EMCYAm5EDhMgfGgOnU0J5\"\n",
    "solution=$(echo $encrypt | openssl enc -md sha256 -aes-128-cbc -a -d -salt -pass pass:$pass1 2> /dev/null)\n",
    "echo \"$solution\"; echo\n",
    "for l in \"$solution\"; do eval \"${l}\"; done"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "### Selected columns and pattern match"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "Suppose we are concerned with only one column of a file. We have information on how the columns are separated and the position of the column.\n",
    "\n",
    "We can do that easily with \"AWK\". It is more than a command, a fully fledged programming language and the Swiss army knife of text processing in Unix! It is mostly used for short scripts and especially very very powerful one-liners.\n",
    "\n",
    "I hope you watched the old AT&T video on Unix pipes, told by a bearded gentleman. Brian Kernighan is one of the most colossal figures in history of computing and the \"K\" in \"AWK\" stands for his surname"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "whatis awk\n",
    "tldr awk"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "But first we have to know the position of a column with a certain heading, such as endYear:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "head -1 $imdbdir/tsv2/title.basics.tsv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "We can easily count that it is the 7th line. But suppose we want to do that programmatically.\n",
    "\n",
    "We need a few more tools"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "whatis grep\n",
    "tldr grep"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "grep is a very important tool to make pattern search in files. With the -n flag, it can return the line number of match\n",
    "\n",
    "However, the input is not separated into lines :(\n",
    "\n",
    "First we have to convert tabs into newlines, with any of the two tools below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "whatis tr\n",
    "tldr tr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "head -1 $imdbdir/tsv2/title.basics.tsv | tr \"\\t\" \"\\n\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "whatis sed\n",
    "tldr sed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "While tr is a more specific tool for transforming characters in a file or stdin, sed is more powerful and versatile\n",
    "\n",
    "It can substitute pattern matches with new strings, extract certain lines, append, delete or insert lines"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "head -1 $imdbdir/tsv2/title.basics.tsv | sed 's/\\t/\\n/g'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "Above sed command reads:\n",
    "\n",
    "- Substitute (s)\n",
    "- tab character (\\t)\n",
    "- with a newline character (\\n)\n",
    "- globally everywhere in the input (g)\n",
    "\n",
    "\"grep\" stands for, Global/Regular Expressions/Print"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "awk is more versatile and handle more complex task but harder to write, sed is only a subset of awk but easier to write and tr is a subset of sed but even easier to use.\n",
    "\n",
    "IMHO, if an easier tool exists for a specific task, I go for it, so I prefer \"tr\" here:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "Now, we should search for endYear and get its line count:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "head -1 $imdbdir/tsv2/title.basics.tsv | tr \"\\t\" \"\\n\" | grep -n endYear"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "That's good it is the 7th column. But programmatically we should only get that 7 and not the rest.\n",
    "\n",
    "We can feed into grep again as such:(don't mind the pattern here)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "head -1 $imdbdir/tsv2/title.basics.tsv | tr \"\\t\" \"\\n\" | grep -n endYear | grep -Po \"^\\d+(?=:)\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "It reads:\n",
    "\n",
    "- Start from the beginning (^)\n",
    "- Get the digits (\\d)\n",
    "- Of 1 or more (+)\n",
    "- Until (?=\n",
    "- The colon(:)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "Regex is a very powerful domain specific language to match patterns in text and as a big data manager you WILL surely need it in many tasks\n",
    "\n",
    "We will return back to regex and make recommendations in later sessions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "Maybe we can get a more concise way to extract the number? We can revert back to sed for this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "head -1 $imdbdir/tsv2/title.basics.tsv | tr \"\\t\" \"\\n\" | sed -n '/endYear/='"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "In grep you cannot return the line number and suppress the match at the same time. But in sed you can only return the line number of match"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "Now in order to use this column position inside awk, we can save it into a variable called CN for \"column number\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "CN=$(head -1 $imdbdir/tsv2/title.basics.tsv | tr \"\\t\" \"\\n\" | sed -n '/endYear/=')\n",
    "echo $CN"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "In more complex code, inserting echo (or printf) statements is important for subsequent debugging"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "So we can save output of any command into a variable with this syntax"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "Getting a column with an index is trivial for awk:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "awk -F \"\\t\" '{ print $7}' $imdbdir/tsv2/title.basics.tsv | head"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "\"-F\" sets the field separator to tab character.\n",
    "\n",
    "But awk has its own variables and we have to pass this CN variable into awk with the -v flag.\n",
    "\n",
    "We can use the same variable name or a different one. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "awk -v CN=$CN -F \"\\t\" '{ print $CN }' $imdbdir/tsv2/title.basics.tsv | head"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "Note that the $CN inside awk expression is different from the one we defined before. Variables of awk are distinct from that of the shell awk is invoked from.\n",
    "\n",
    "Combined steps are:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "CN=$(head -1 $imdbdir/tsv2/title.basics.tsv | tr \"\\t\" \"\\n\" | sed -n '/endYear/=')\n",
    "awk -v CN=$CN -F \"\\t\" '{ print $CN }' $imdbdir/tsv2/title.basics.tsv | head"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "Or in a single statement:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "awk -v CN=$(head -1 $imdbdir/tsv2/title.basics.tsv | tr \"\\t\" \"\\n\" | sed -n '/endYear/=') \\\n",
    "-F \"\\t\" '{ print $CN }' $imdbdir/tsv2/title.basics.tsv | head"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "Writing strictly one-liner codes is not the best skill a coder should have.\n",
    "\n",
    "Writing a code that is easily understandable by the coder h(er|im)self and collaborators and easily debuggable is a more important skill\n",
    "\n",
    "So I go for the two-liner version"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "But wait .... I refer to the filename twice! Suppose I had to do that many more times, referring to the same path again and again. Wouldn't it be better to save it into a variable and reuse? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "filename=$imdbdir/tsv2/title.basics.tsv\n",
    "CN=$(head -1 $filename | tr \"\\t\" \"\\n\" | sed -n '/endYear/=')\n",
    "awk -v CN=$CN -F \"\\t\" '{ print $CN }' $filename | head"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "But for a more reusable code, we can also parametrize the pattern and the last command to get the head or tail"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "filename=$imdbdir/tsv2/title.basics.tsv\n",
    "pattern=endYear\n",
    "comnd=head\n",
    "CN=$(head -1 $filename | tr \"\\t\" \"\\n\" | sed -n \"/$pattern/=\")\n",
    "awk -v CN=$CN -F \"\\t\" '{ print $CN }' $filename | $comnd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "And we can wrap all inside a shell function with parameters to be passed.\n",
    "\n",
    "Suppose first parameter is the filename, second is the pattern and the last is the command (head or tail)\n",
    "\n",
    "Parameters passed to the function can be referred by their positions as $1, $2 ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "# $imdbdir/tsv2/title.basics.tsv\n",
    "# endYear\n",
    "# head\n",
    "\n",
    "headortail()\n",
    "{\n",
    "    filename=$1\n",
    "    pattern=$2\n",
    "    comnd=$3\n",
    "    CN=$(head -1 $filename | tr \"\\t\" \"\\n\" | sed -n \"/$pattern/=\")\n",
    "    awk -v CN=$CN -F \"\\t\" '{ print $CN }' $filename | $comnd\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "Now we can reuse this code for many similar purposes. Let's start with our initial intent:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "headortail $imdbdir/tsv2/title.basics.tsv endYear head"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "And get the last values of primaryName column from the name.basics.tsv file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "headortail $imdbdir/tsv2/name.basics.tsv primaryName tail"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "Note that we can also do the pattern match with awk, and more powerful than sed or grep:\n",
    "\n",
    "For example let's return the whole line where 7th field matches \"endYear\", prefixed with line number"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "filename=$imdbdir/tsv2/title.basics.tsv\n",
    "awk -F \"\\t\" '$7 == \"endYear\" { print NR\": \"$0 }' $filename"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "**EXERCISE 9:**\n",
    "\n",
    "Write a function named linepattern that takes a file, a column pattern, a row pattern and returns the line number of match and the whole line\n",
    "\n",
    "Note that you can pass multiple variables to awk by invking \"-v\" multiple times for each variable\n",
    "\n",
    "And string variables with whitespaces must be wrapped inside quotes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "**SOLUTION 9:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "pass1=\n",
    "encrypt=\"U2FsdGVkX18yoFtMKddz+N0MjU6twanensOn3zp1K/Bkv7lPtruYc23Ma6LoOjGk dlozrWW+rG3dgDq6TOYcgrpO1d4pIZomvyE01bWSDh1WXgXx0s+wTt8EPAIj/rg7 B5Nqcy4U/24rXJq59n6IkAOFrFFKCcDK3Z/D62FxI5al0tH5cyQVkWv+7NJSf+T7 Ctl3wN3Cjt4kHZXkL2YX7PfznQwOJb34BvWUd5Vl5XxTKmrbuS801+41XQdWUVJ5 kEXlxR3nmL3WPvLtfT/0Ux1VLjzJxzFtaUmvyDGigdoHIp/qXS+jv+I4pmXG8GWs\"\n",
    "solution=$(echo $encrypt | openssl enc -md sha256 -aes-128-cbc -a -d -salt -pass pass:$pass1 2> /dev/null)\n",
    "echo \"$solution\"; echo\n",
    "for l in \"$solution\"; do eval \"${l}\"; done"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "Let's call it for name.basics.tsv, primaryName and \"Al Pacino\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "linepattern $imdbdir/tsv2/name.basics.tsv primaryName \"Al Pacino\" | column -t"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "### Unique values of a field"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "Now out of millions of rows, let's explore the unique values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "whatis sort\n",
    "tldr sort"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "whatis uniq\n",
    "tldr uniq"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "It is a coincidence that UNIQ is just next to our building!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "filename=$imdbdir/tsv2/title.basics.tsv\n",
    "pattern=endYear\n",
    "CN=$(head -1 $filename | tr \"\\t\" \"\\n\" | sed -n \"/$pattern/=\")\n",
    "awk -v CN=$CN -F \"\\t\" '{ print $CN }' $filename | sort | uniq"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "In order for \"uniq\" to work, stdin must be well sorted. But wait, sort command can also do uniq with \"-u\" flag:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "filename=$imdbdir/tsv2/title.basics.tsv\n",
    "pattern=endYear\n",
    "CN=$(head -1 $filename | tr \"\\t\" \"\\n\" | sed -n \"/$pattern/=\")\n",
    "awk -v CN=$CN -F \"\\t\" '{ print $CN }' $filename | sort -u"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "But for single column values, that tall of an output does not look good. We can get it arranged and printed in columns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "whatis pr\n",
    "man pr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "filename=$imdbdir/tsv2/title.basics.tsv\n",
    "pattern=endYear\n",
    "CN=$(head -1 $filename | tr \"\\t\" \"\\n\" | sed -n \"/$pattern/=\")\n",
    "awk -v CN=$CN -F \"\\t\" '{ print $CN }' $filename | \\\n",
    "sort -u | pr -8 -t"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "**EXERCISE 10:**\n",
    "\n",
    "Get the unique values of startYear column of title.basics.tsv but report only year values (just numbers)\n",
    "\n",
    "You can reuse components from previous grep commands for regex"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "**SOLUTION 10:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "pass1=\n",
    "encrypt=\"U2FsdGVkX19ZYi1B9oGWAmWy5lxS4X1HuF6mo6QMpjWeoHS34ZDvXidi0RTD1WpE wLZD8MkYgZmlD+t4jFS9iTtrrd5zXO74cehaQUVvquHcfFyI2F8MZOQpxvHdwEFj QP96UYpY9c+a/E/SuDbPEldCMrDS8Q/haRoaUKn4JWI51bbVhG3o0vwNjgVmBGKW xLYOEK/FHiVpKcuxb2u0THLrKquzIFddJkkOYXqLpjYQ8yxiL27D6eMLlcpc/+aQ QQObeW1jV5D0p7ewvG85Px6w0U9R3SFAA27RUnq+wmY=\"\n",
    "solution=$(echo $encrypt | openssl enc -md sha256 -aes-128-cbc -a -d -salt -pass pass:$pass1 2> /dev/null)\n",
    "echo \"$solution\"; echo\n",
    "for l in \"$solution\"; do eval \"${l}\"; done"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "The dataset stretches far back!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "Now let's say the counts of unique values are also of importance:\n",
    "\n",
    "Sort in reverse order in terms of counts (only years)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "counts=$(filename=$imdbdir/tsv2/title.basics.tsv\n",
    "pattern=startYear\n",
    "CN=$(head -1 $filename | tr \"\\t\" \"\\n\" | sed -n \"/$pattern/=\")\n",
    "awk -v CN=$CN -F \"\\t\" '{ print $CN }' $filename | \\\n",
    "grep -Po \"\\d+\" | sort | uniq -c)\n",
    "\n",
    "echo \"$counts\" | sort -nr | column -c 100"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "And let's sort in increasing year order:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "echo \"$counts\" | sort -n -k 2 | column -c 100"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "And how about a simple plot wihout leaving the terminal/shell?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "echo \"$counts\" | sort -n -k 2 | \\\n",
    "gnuplot -e \"set terminal dumb; plot '-' using 2:1 w points pt '*'\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "And let's just focus on the period between 1980 and 2017:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "echo \"$counts\" | sort -n -k 2 | awk -F \" \" '$2 <= 2017 && $2 >= 1980' | \\\n",
    "gnuplot -e \"set terminal dumb; plot '-' using 2:1 w points pt '*'\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "### Multiple filters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "Suppose we use the awk code for filtering on multiple criteria:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "filename=$imdbdir/tsv2/title.basics.tsv\n",
    "head -1 $filename"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "Let's exclude first two columns, keep the header, filter for matching Godfather in primary Title and startYear after 1972 and before 1990:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "CN1=$(head -1 $filename | tr \"\\t\" \"\\n\" | sed -n \"/primaryTitle/=\")\n",
    "CN2=$(head -1 $filename | tr \"\\t\" \"\\n\" | sed -n \"/startYear/=\")\n",
    "{ head -1 $filename | awk -F \"\\t\" 'BEGIN {OFS = \"\\t\"} { $1=$2=\"\";print $0 }'; \\\n",
    "awk -v CN1=$CN1 -v CN2=$CN2 -F \"\\t\" \\\n",
    "'BEGIN {OFS = \"\\t\"} $CN1 ~ /Godfather/ && $CN2 > 1972 && $CN2 < 1990 { $1=$2=\"\";print $0 }' $filename; } | \\\n",
    "column -t -s $'\\t'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "OFS is the output field separator, in order to format better for column command\n",
    "\n",
    "We delete first two columns by setting them to empty string\n",
    "\n",
    "And we combine both the header and filtered rows with { line1; line2; } | further_commands, to format together with column\n",
    "\n",
    "We combine multiple filters with &&"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "**EXERCISE 11:**\n",
    "\n",
    "Delete first 2 and isAdult columns, keep the header\n",
    "Filter for movies that have a runtime less than 10 minutes and not \\N, genres include thriller and start year is after 1990 and not \\N\n",
    "Note that you have to escape the escape character in \"\\N\" as such: \"\\\\N\"\n",
    "\n",
    "Run head -1 for once and save in a variable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "**SOLUTION 11:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "pass1=\n",
    "encrypt=\"U2FsdGVkX1+S79PKqpQJiVEf5D6Y/snYFkTIjRPI+v/ZbkuGO+wB6rmkGH5AHHUg 7aEK+wO93DX02gFwYjkltP3OTIxg7wxJFjAfRYGMUriQK/DbcPMqlWRyAZWp2pXo vrI0smeerjV5d4wyMQDe33lYvbfFq6sC1cZvbmbQeOxv4ET8O+BapcDVt4ULXYC+ B75Uc1g0PrqM9fewr0IqQX+xCaqAAn4j8ly5nxhxVGgneTrErUeO6QP9oEx1FCvz vFXhCdm1uyqKtBOO8GB/OUQ5Mz6PvlNM87RNNuNhmrhnsjwqw97eC8uLQKssASor mi/5t9NebnQgHAf0IShq0L/KawPfnm/KmYXjb2DqSSu+9OM1ZyxpDAgjveD/TNbR C0Re7OgVfi6kg22LiZpRiHks4+clqnkjF6cCA54dFu5Qf49orXJNtB7HOakorRoD 0puOPrWRy754JLxLzgKnnE7B6+OJ99uFWQGLXuT5RKfdtBc6wWjXZkSNPVoK8vTV cihK+YKQum03N/kkZ6cxtTRnT+ZCG5zyZBOFxOS+w0qt9NeYd1wbG9thr/CSvkzJ dgycZl31MeOXzqUatZsQg9x13Aq6NVBusM8u1l2fw6bFCDlmSHdHyEp+UsLhfaTK GvDyAy8VI2AZQDIWTDoxbU6DdLdrWgsUT7TkImjFjn7WCFpRqS6OAAyYBEYfuko9 CH0VWfwJYiCs3O2HXd3p/V/aqPPs1z9ZitI+IvzodAjLRC2+YST5t3QKxVBKsf1y wqTuDpOOHMXmIYjChKD/6vfalCYxD7uza+DKVoPCUX0=\"\n",
    "solution=$(echo $encrypt | openssl enc -md sha256 -aes-128-cbc -a -d -salt -pass pass:$pass1 2> /dev/null)\n",
    "echo \"$solution\"; echo\n",
    "for l in \"$solution\"; do eval \"${l}\"; done"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "Even very strict filters can yield results!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "For more complex queries involving joins among tables is beyond the intent of awk language. And referring to even more column names complicates the code.\n",
    "\n",
    "Now we need to switch to SQL\n",
    "\n",
    "But first we have to process the data to better fit PostgreSQL database server"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "# Data wrangling with shell and R"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "## Split fields and melt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "Let's remember the fields of files again:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "find $imdbdir/tsv2 -mindepth 1 | \\\n",
    "    parallel -k -j0 'printf \"%s\\n\" {}; head -2 {} | column -t; echo'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "See that the field that starts with \"\"tt\" appears in all but one of the files. This field is the unique code for each title\n",
    "\n",
    "And the field that starts with \"nm\" appears in title.principals, title.crew and name.basics and is the unique code for each person in movies (actor, director, or any profession)\n",
    "\n",
    "We can join the tables based on these common fields to create more complex results\n",
    "\n",
    "But we have a problem! The principalCast field in title.principals file combine all principal people of a move into a separate field with commas!\n",
    "\n",
    "We would expect a separate row for each unique combination of a title and a person\n",
    "\n",
    "Each tool has its own strengths. shell was powerful until now, but we need something specifically designed to transform data, R!\n",
    "\n",
    "data.table is an expressive, efficient and very powerful package designed for handling very large datasets and with a syntax with components that corresponds to SQL clauses:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "**When the value at the dropdown box on the right shows \"R\", it means, now we are sending our commands to the R interpreter, not Bash**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "First load the library:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "ir"
   },
   "outputs": [],
   "source": [
    "library(data.table)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "Set the main directory path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "ir"
   },
   "outputs": [],
   "source": [
    "prefix <- \"~/data/imdb\"\n",
    "prefix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "ir"
   },
   "source": [
    "And the file path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "ir"
   },
   "outputs": [],
   "source": [
    "in_path <- sprintf(\"%s/tsv2/title.principals.tsv\", prefix)\n",
    "in_path"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "ir"
   },
   "source": [
    "Fast read the file into R using data.table:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "ir"
   },
   "outputs": [],
   "source": [
    "principals <- data.table::fread(in_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "ir"
   },
   "source": [
    "And split and melt from the second column. Note that since the resulting object is +25m lines, we here limit the processed lines in order to save time on binder:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "ir"
   },
   "outputs": [],
   "source": [
    " principals_molten <- principals[1:1e5, {\n",
    "    principalCast_list = strsplit(principalCast, \",\");\n",
    "    .(\n",
    "        tconst = rep(tconst, sapply(principalCast_list, length)),\n",
    "        principalCast = unlist(principalCast_list)\n",
    "    )\n",
    " }\n",
    "]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "ir"
   },
   "source": [
    "What does it do?\n",
    "\n",
    "```r\n",
    "principalCast_list = strsplit(principalCast, \",\");\n",
    "```\n",
    "\n",
    "creates an intermadiate object that is an R list, each item of which is a character vector created from each original row by splitting second field from commas\n",
    "\n",
    "```r\n",
    "tconst = rep(tconst, sapply(principalCast_list, length))\n",
    "```\n",
    "\n",
    "for each lish item, replicates the tconst field with the count of principalCast - one for each person\n",
    "\n",
    "```r\n",
    "principalCast = unlist(principalCast_list)\n",
    "```\n",
    "\n",
    "Now that we have a vector of tconst - title code - that is replicated enough for the number of principal cast in a title, we can combine all list items into a long vector"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "ir"
   },
   "source": [
    "Let's see in action using only the first two lines:\n",
    "\n",
    "In the first (second) row there are three (two) principal cast concatenated with commas: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "ir"
   },
   "outputs": [],
   "source": [
    "principals12 <- principals[1:2]\n",
    "principals12"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "ir"
   },
   "source": [
    "Now we split each row of second column from commas:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "ir"
   },
   "outputs": [],
   "source": [
    "principalCast_list = strsplit(principals12[,principalCast], \",\")\n",
    "principalCast_list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "ir"
   },
   "source": [
    "To view the list better:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "ir"
   },
   "outputs": [],
   "source": [
    "listviewer::jsonedit(principalCast_list, mode = \"form\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "ir"
   },
   "source": [
    "We want to end up with 5 rows: 3 + 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "ir"
   },
   "outputs": [],
   "source": [
    "lengths <- sapply(principalCast_list, length)\n",
    "lengths"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "ir"
   },
   "source": [
    "In the first title, there are 3 principal cast and in the second title 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "ir"
   },
   "outputs": [],
   "source": [
    "tconst <- rep(principals12[,tconst], lengths)\n",
    "tconst"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "ir"
   },
   "source": [
    "Now the title codes are replicated with relavant counts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "ir"
   },
   "outputs": [],
   "source": [
    "principalCast <- unlist(principalCast_list)\n",
    "principalCast"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "ir"
   },
   "source": [
    "And the list is simplified into a single vector. tconst and principalCast are both of length 5, thus we can combine them together"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "ir"
   },
   "outputs": [],
   "source": [
    "combined <- list(tconst = tconst, principalCast = principalCast)\n",
    "listviewer::jsonedit(combined, mode = \"form\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "ir"
   },
   "source": [
    "And a list of equal length vectors can be converted to a data.frame or better a data.table:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "ir"
   },
   "outputs": [],
   "source": [
    "principals12_molten <- as.data.table(combined)\n",
    "principals12_molten"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "ir"
   },
   "source": [
    "Compare initial and final objects:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "ir"
   },
   "outputs": [],
   "source": [
    "principals12\n",
    "principals12_molten"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "ir"
   },
   "outputs": [],
   "source": [
    "lapply(list(principals[1:1e5], principals_molten), dim)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "ir"
   },
   "source": [
    "If we had run the code in the whole dataset, we would have started with a data.table of 4 million rows and have ended up with a data.table of 25 million rows in a matter of few seconds in powerful enough computer!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "ir"
   },
   "source": [
    "The single data.table statement combines all these steps and repeats for all rows. The initial and final dimensions are:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "ir"
   },
   "source": [
    "Now create a filename for output:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "ir"
   },
   "outputs": [],
   "source": [
    "out_path <- sprintf(\"%s/tsv2/title.principals_melt_short.tsv\", prefix)\n",
    "out_path"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "ir"
   },
   "source": [
    "And fast write the data.table into a tsv file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "ir"
   },
   "outputs": [],
   "source": [
    "fwrite(principals_molten, file = out_path, sep = \"\\t\") # fast write new DT as tsv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "ir"
   },
   "source": [
    "Now back to bash, let's view the new data file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "head -20 $imdbdir/tsv2/title.principals_melt_short.tsv | column -t"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "Bonus: Below parallel code does the same thing somwhow far less efficiently, so don't try it (while loop is there for you not to execute the code). This is here for demonstration purposes only. You may skip it.\n",
    "\n",
    "R code is far less concise and efficient"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "while false;\n",
    "do\n",
    "head -1 $imdbdir/tsv2/title.principals.tsv; tail -n+2 $imdbdir/tsv2/title.principals.tsv | \\\n",
    "parallel -j0 \"line={}; \\\n",
    "    tc=\\${line%%$'\\t'*}; \\\n",
    "    nm=\\${line##*$'\\t'}; \\\n",
    "    nml=\\${nm//,/$'\\n'}; \\\n",
    "    ln=\\$(echo -n \\\"\\$nml\\\" | grep -c ^); \\\n",
    "    paste <(yes \\$tc | head -n \\$ln) <(echo -n \\\"\\$nml\\\") --delimiters \\\"\\t\\\"\" \\\n",
    "    > $imdbdir/tsv2/title.principals_melt_shell.tsv\n",
    "done"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "Inside the command to be executed by parallel:\n",
    "\n",
    "- ```{}``` stands for input\n",
    "- ```line={};``` assigns the input into $line variable\n",
    "- ```tc=\\${line%%$'\\t'*}``` deletes the part including and after the tab character, so gets only the tconst portion. It is a more advanced topic in shell scripting called \"parameter substitution\"\n",
    "- ```nm=\\${line##*$'\\t'}``` deletes the part including and before the tab character so only leaves the nm part. Again parameter substitution\n",
    "- ```nml=\\${nm//,/$'\\n'};``` replaces the commas between nm's by newlines, so practically splits them into separate lines\n",
    "- ```ln=\\$(echo -n \\\"\\$nml\\\" | grep -c ^)``` counts the number of lines of nm's, so number of times tconst should be repeated\n",
    "- ```yes \\$tc | head -n \\$ln``` repeats the tconst the exact number of times equal to the number of nm's\n",
    "- ```paste <(yes \\$tc | head -n \\$ln) <(echo -n \\\"\\$nml\\\") --delimiters \\\"\\t\\\"``` combines two columns of tc and nm into a tab separated text. This line utilizes a more advanced topic in shell scripting called \"process substitution\" "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "For subsequent codes, it is better that we clean the memory allocated to R for better performance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "ir"
   },
   "outputs": [],
   "source": [
    "rm(list =ls())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "**EXERCISE 12:**\n",
    "\n",
    "In R, create a file title.genres_short.tsv under tsv2 in which there are two columns from title.basics.tsv, tconst and genres: genres split from commas and tconst replicated accordingly. Take only the first 1e5 (100k) lines. Compare the dimension of input and output objects/files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "ir"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "ir"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "ir"
   },
   "source": [
    "**SOLUTION 12:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "ir"
   },
   "outputs": [],
   "source": [
    "pass <- readline(prompt = \"Please enter the password for the solution: \")\n",
    "encrypt <- \"U2FsdGVkX1+bbnopeIKcv3xn7U0pmH8WCpExJzguv8sSJBF+o0ipnhkCqyXVydjb 6RSgGNjWN/dEVp9Fq2Ht/B8v2rWKSEyOwtlfBQ0l4ecTRBGYWmK0AQRINUaTIXZS 2MCRKFA0fxjp9bENNr1hcldd68IcWYSLLnzVto7TFrBD4rmSSGXyedb4+9VBQafX FaY1C0EHuGsA0YW7/F5QXrv164or/Shm6+Zi1aaEpnl0HHviHdC1n5hXeiIyD6g7 iEHCsdVm68V1L1SI3ALHbf9nOnV56+Hgp59I5slx0N/DMmLtUumQ9pk8A0Lw8mAk 1ojJp2QzG5r89iZ58fVtxj5VDjr/pnQSKHBzfH0GRF+JHIKS90OAmcKzHh/Two6q Ckv0e101B5cargbkk5EKRwYmNqUzTBIzIqqCrqhzbGJS/z9maAHSmSTgqyFYxLnR QgMvuCfGHNiq0O+WR5F7piDVdmpHzG59yD4A/2lZMSe01jppyBXjW/+E5PocIie/ L9E3wxl/ObiRGc6dwCQ1W4DiaEqZJxUpraqANWLJ8Yc7dsPEOB4GBuZnOYP9WGYR sNdJ91nqhJb5eFmbzcOQRBJzEt4onc0dqBdZC3qZTTJzJ7WLShIwr35tOkcrtDxn zbHuFR7j4x4cUVHZcA3UVlG5QrKegLh64jA0UGjUML+Aq5hzjaCcJ+GQeOjkMjHk iLEgMvz+YdFuo9txUm9J8PlR+iOOLz5yI8xNU5YVLZmYvzXFSnGYwQKoAINmlY/7 5T0kAOYVi8HHtgNedmp0Ivs4i5hYPHwbKRDJ5Eh7ews=\"\n",
    "solution <- system(sprintf(\"echo %s | openssl enc -md sha256 -aes-128-cbc -a -d -salt -pass pass:%s 2> /dev/null\", encrypt, pass), intern = T, ignore.stderr = T)\n",
    "cat(solution, sep = \"\\n\")\n",
    "eval(parse(text = solution))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "ir"
   },
   "source": [
    "Let's view the output file from Bash:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "head -20 $imdbdir/tsv2/title.genres_short.tsv | column -t"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "## trim headers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "Now, in order to import tsv's into postgresql, we need to trim the header rows. In csv's, this step is not necessary\n",
    "\n",
    "We will use tail for this purpose, let's see.."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "cat $imdbdir/tsv2/title.basics.tsv | wc -l | numfmt --to=si"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "title.basics file has 4.6 million rows! More than any gui text editor or spreadsheet can handle "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "File starts like that:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "head $imdbdir/tsv2/title.basics.tsv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "And after tail:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "tail -n+2 $imdbdir/tsv2/title.basics.tsv | head"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "We can redirect it to a new file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "mkdir -p $imdbdir/tsv3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "tail -n+2 $imdbdir/tsv2/title.basics.tsv > $imdbdir/tsv3/title.basics2.tsv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "Now do it for all files:\n",
    "\n",
    "if clause checks for existing files, if exists, skips\n",
    "\n",
    "basename extracts the filename\n",
    "\n",
    "${basenm/.tsv/2.tsv} adds a 2 before .tsv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "find $imdbdir/tsv2 -mindepth 1 | \\\n",
    "    parallel -k -j0 \"basenm=\\$(basename {});\n",
    "        if [ ! -e ${imdbdir}/tsv3/\\${basenm/.tsv/2.tsv} ];\n",
    "        then\n",
    "            tail -n+2 {} > \\\n",
    "            ${imdbdir}/tsv3/\\${basenm/.tsv/2.tsv};\n",
    "        fi\n",
    "        \""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "Now it is time to create a database on PostgreSQL server and import the data into!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "# Some other useful \\*nix utilities"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "There are some other utilities that you might find useful while working on \\*nix systems (Linux, MacOS, FreeBSD, even Android)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "To get details on ongoing processes on the operating system:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "whatis ps\n",
    "tldr ps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "For example, list \"python\" processes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "ps aux | grep python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "Get information on the hostname of the computer you are logged into:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "whatis hostname\n",
    "tldr hostname"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "hostname"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "\"ls\" does not report the recursive size of directories, for this you may use du (disk usage)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "whatis du\n",
    "tldr du"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "du -sh $imdbdir/*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "To get information on mounted file systems, use df:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "whatis df\n",
    "tldr df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "To list mounted file systems, with file types and human readable sizes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "df -hT"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "To mount/unmount drives on the root filesystem:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "whatis mount\n",
    "tldr mount"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "whatis umount\n",
    "tldr umount"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "To check the integrity of a filesystem:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "whatis fsck\n",
    "tldr fsck"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "To get superuser privileges on current user (if allowed to) or execute a command as another user:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "whatis sudo\n",
    "tldr sudo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "And to get a dynamic overview of the system resources, utilization and processes: top (by default) or better htop (to be installed separately)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "whatis htop\n",
    "tldr htop"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "Let's open htop on our terminal (press q to quit):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "scr=$(screen -ls | grep -P \"Attached\" | head -1 | grep -Po \"^\\t+.+?(?=\\s|\\t)\" | tr -d \"\\t\")\n",
    "screen -S $scr -X stuff 'htop\\n'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "For a very versatile, efficient, lightweight and extensible text and code editor, vim is the tools of power users:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "whatis vim\n",
    "tldr vim"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "- Enter edit mode with \"i\",\n",
    "- Exit edit mode and enter into command mode with \"Esc\",\n",
    "- To save \":w\",\n",
    "- To quit after save \":q\",\n",
    "- To quit without save \":q!\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "scr=$(screen -ls | grep -P \"Attached\" | head -1 | grep -Po \"^\\t+.+?(?=\\s|\\t)\" | tr -d \"\\t\")\n",
    "screen -S $scr -X stuff 'vim\\n'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "There are also many networking tools in \\*unix that make it easier and efficient to script and automatize networking tasks and monitor networking activities such as:\n",
    "\n",
    "ip, ifconfig, route, netstat, nmap, ping, bwm-ng, nethogs, iftop"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "To get and parse data from the network useful tools are:\n",
    "\n",
    "curl, wget, lynx, w3m, elinks. You can also use chrome headless from the command line"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "# Bonus homework 01"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "In order to induce you to \"dirty your hands\" with the shell tools, here is a little assignment that will count as additional 5 points if done right:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "Great \"sed\" tool has a simple tool to extract any arbitrary lines by line number from a text stream as such:\n",
    "\n",
    "For example, extract the 10th to 20th lines from input:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "seq 100 | sed -n \"10,20p\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "sed can both read from stdin (input from the pipe) and a file as the last parameter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "tldr sed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "While seq creates a sequence of numbers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "whatis seq\n",
    "tldr seq"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "Your task is to write a wrapper function called windowsearch take takes four parameters:\n",
    "- First parameter is the starting line number for the window\n",
    "- Second parameter is the ending line number for the window\n",
    "- Third parameter is the position of sort field\n",
    "- Fourth parameter is the filename\n",
    "- The function takes the three parameters and using sed, returns those lines in between the two line numbers from the file by sorting according to the nth field as specified by the third parameter\n",
    "- So sed output should pipe into sort\n",
    "- Note that positional parameters are referred to by their position as \\\\$1, \\\\$2\n",
    "- You may or may not assign those positional parameters into named variables inside the function. In a simple context it is not too important. In more complex scripts, we may lose the track of what \\\\$1, \\\\$2 and so on stands for "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "**SOLUTION BONUS 1:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "pass1=\n",
    "encrypt=\"U2FsdGVkX1/Y9ExlbpCUNWw72NzPn6D9LH0B3yzGkj3GMYpzObI/hn2Tq+EE+Iy3 7jJWVnuYZf1v4/e4tvoV/UaFzwtrXYmJOdUj86dm93vC5yTkzzL13USrujOKSwGD mFM+ZWikVnFmGWHBi+NaCvbrQ/0Bl+96dmp1Z7zPv6JvApA5DWLBNarG8il+Oc+3 fRdUkV0BscSY9GoWWloMTj/QvCOdfG7xfIhCe4JSCkI=\"\n",
    "solution=$(echo $encrypt | openssl enc -md sha256 -aes-128-cbc -a -d -salt -pass pass:$pass1 2> /dev/null)\n",
    "echo \"$solution\"; echo\n",
    "for l in \"$solution\"; do eval \"${l}\"; done"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "To check your function, invoking it with the following parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "kernel": "bash"
   },
   "outputs": [],
   "source": [
    "windowsearch 10 20 2 $imdbdir/tsv2/title.principals.tsv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "should yield:\n",
    "```\n",
    "tt0000009\tnm0085156,nm0063086,nm1309758,nm0183823\n",
    "tt0000014\tnm0166380,nm0525910,nm0244989\n",
    "tt0000010\tnm0525910\n",
    "tt0000016\tnm0525910\n",
    "tt0000012\tnm0525910,nm0525908\n",
    "tt0000015\tnm0721526\n",
    "tt0000018\tnm0804434,nm3692071\n",
    "tt0000019\tnm0932055\n",
    "tt0000013\tnm1715062,nm0525910,nm0525908\n",
    "tt0000017\tnm3691272,nm0804434,nm1587194,nm3692829\n",
    "tt0000011\tnm3692297,nm0804434\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "kernel": "bash"
   },
   "source": [
    "Following the instructions in the instructor message on how to submit your solution"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "SoS",
   "language": "sos",
   "name": "sos"
  },
  "language_info": {
   "codemirror_mode": "sos",
   "file_extension": ".sos",
   "mimetype": "text/x-sos",
   "name": "sos",
   "nbconvert_exporter": "sos_notebook.converter.SoS_Exporter",
   "pygments_lexer": "sos"
  },
  "sos": {
   "kernels": [
    [
     "Bash",
     "bash",
     "Bash",
     "#E6EEFF"
    ],
    [
     "Bash",
     "bash",
     "",
     "#E6EEFF"
    ],
    [
     "R",
     "ir",
     "",
     "#DCDCDA"
    ]
   ],
   "panel": {
    "displayed": true,
    "height": 0,
    "style": "side"
   },
   "version": "0.19.18"
  },
  "toc": {
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "toc_cell": false,
   "toc_position": {
    "height": "705.696px",
    "left": "0px",
    "right": "1434px",
    "top": "110.284px",
    "width": "477.983px"
   },
   "toc_section_display": "block",
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}