etl.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true
   },
   "source": [
    "# ETL Processes\n",
    "Use this notebook to develop the ETL process for each of your tables before completing the `etl.py` file to load the whole datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import glob\n",
    "import psycopg2\n",
    "import pandas as pd\n",
    "from sql_queries import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true
   },
   "outputs": [],
   "source": [
    "from create_tables import create_database\n",
    "create_database()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true
   },
   "outputs": [],
   "source": [
    "conn = psycopg2.connect(\"host=127.0.0.1 dbname=sparkifydb user=student password=student\")\n",
    "cur = conn.cursor()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true
   },
   "outputs": [],
   "source": [
    "def get_files(filepath):\n",
    "    all_files = []\n",
    "    for root, dirs, files in os.walk(filepath):\n",
    "        files = glob.glob(os.path.join(root,'*.json'))\n",
    "        for f in files :\n",
    "            all_files.append(os.path.abspath(f))\n",
    "    \n",
    "    return all_files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true
   },
   "source": [
    "# Process `song_data`\n",
    "In this first part, you'll perform ETL on the first dataset, `song_data`, to create the `songs` and `artists` dimensional tables.\n",
    "\n",
    "Let's perform ETL on a single song file and load a single record into each table to start.\n",
    "- Use the `get_files` function provided above to get a list of all song JSON files in `data/song_data`\n",
    "- Select the first song in this list\n",
    "- Read the song file and view the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true
   },
   "outputs": [],
   "source": [
    "song_files = get_files(\"data/song_data\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true
   },
   "outputs": [],
   "source": [
    "filepath = song_files[0]\n",
    "filepath"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true
   },
   "outputs": [],
   "source": [
    "df = pd.read_json(filepath, lines=True)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true
   },
   "source": [
    "## #1: `songs` Table\n",
    "#### Extract Data for Songs Table\n",
    "- Select columns for song ID, title, artist ID, year, and duration\n",
    "- Use `df.values` to select just the values from the dataframe\n",
    "- Index to select the first (only) record in the dataframe\n",
    "- Convert the array to a list and set it to `song_data`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true
   },
   "outputs": [],
   "source": [
    "song_data = df[[\"song_id\", \"title\", \"artist_id\", \"year\", \"duration\"]].values.tolist()[0]\n",
    "song_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true
   },
   "source": [
    "#### Insert Record into Song Table\n",
    "Implement the `song_table_insert` query in `sql_queries.py` and run the cell below to insert a record for this song into the `songs` table. Remember to run `create_tables.py` before running the cell below to ensure you've created/resetted the `songs` table in the sparkify database."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true
   },
   "outputs": [],
   "source": [
    "cur.close()\n",
    "conn.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true
   },
   "outputs": [],
   "source": [
    "cur.execute(\"\"\"\n",
    "    drop table songs;\n",
    "\"\"\")\n",
    "conn.commit()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true
   },
   "outputs": [],
   "source": [
    "cur.execute(\"\"\"\n",
    "create table if not exists songs(song_id varchar, title varchar not null, artist_id varchar, year int, duration decimal not null, primary key (song_id))\n",
    "\"\"\")\n",
    "conn.commit()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true
   },
   "outputs": [],
   "source": [
    "cur.execute(\"\"\"\n",
    "    insert into songs(song_id, title, artist_id, year, duration) values(%s, %s, %s, %s, %s) on conflict do nothing\n",
    "\"\"\", song_data)\n",
    "conn.commit()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true
   },
   "outputs": [],
   "source": [
    "cur.execute(\"\"\"\n",
    "select * from songs;\n",
    "\"\"\")\n",
    "conn.commit()\n",
    "row = cur.fetchone()\n",
    "while row:\n",
    "   print(row)\n",
    "   row = cur.fetchone()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true
   },
   "source": [
    "Run `test.ipynb` to see if you've successfully added a record to this table."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true
   },
   "source": [
    "## #2: `artists` Table\n",
    "#### Extract Data for Artists Table\n",
    "- Select columns for artist ID, name, location, latitude, and longitude\n",
    "- Use `df.values` to select just the values from the dataframe\n",
    "- Index to select the first (only) record in the dataframe\n",
    "- Convert the array to a list and set it to `artist_data`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true
   },
   "outputs": [],
   "source": [
    "artist_data = df[[\"artist_id\", \"artist_name\", \"artist_location\", \"artist_latitude\", \"artist_longitude\"]].values.tolist()[0]\n",
    "artist_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true
   },
   "source": [
    "#### Insert Record into Artist Table\n",
    "Implement the `artist_table_insert` query in `sql_queries.py` and run the cell below to insert a record for this song's artist into the `artists` table. Remember to run `create_tables.py` before running the cell below to ensure you've created/resetted the `artists` table in the sparkify database."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true
   },
   "outputs": [],
   "source": [
    "cur.execute(\"\"\"\n",
    "drop table artists\n",
    "\"\"\")\n",
    "conn.commit()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true
   },
   "outputs": [],
   "source": [
    "cur.execute(\"\"\"\n",
    "create table if not exists artists(artist_id varchar, name varchar not null, location varchar, latitude float, longitude float, primary key (artist_id))\n",
    "\"\"\")\n",
    "conn.commit()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true
   },
   "outputs": [],
   "source": [
    "cur.execute(\"\"\"\n",
    "insert into artists(artist_id, name, location, latitude, longitude) values (%s, %s, %s, %s, %s) on conflict do nothing\n",
    "\"\"\", artist_data)\n",
    "conn.commit()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true
   },
   "outputs": [],
   "source": [
    "cur.execute(\"\"\"\n",
    "select * from artists;\n",
    "\"\"\")\n",
    "conn.commit()\n",
    "row = cur.fetchone()\n",
    "while row:\n",
    "   print(row)\n",
    "   row = cur.fetchone()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true
   },
   "source": [
    "Run `test.ipynb` to see if you've successfully added a record to this table."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true
   },
   "source": [
    "# Process `log_data`\n",
    "In this part, you'll perform ETL on the second dataset, `log_data`, to create the `time` and `users` dimensional tables, as well as the `songplays` fact table.\n",
    "\n",
    "Let's perform ETL on a single log file and load a single record into each table.\n",
    "- Use the `get_files` function provided above to get a list of all log JSON files in `data/log_data`\n",
    "- Select the first log file in this list\n",
    "- Read the log file and view the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true
   },
   "outputs": [],
   "source": [
    "log_files = get_files(\"data/log_data\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true
   },
   "outputs": [],
   "source": [
    "filepath = log_files[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true
   },
   "outputs": [],
   "source": [
    "df = pd.read_json(filepath, lines=True)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true
   },
   "source": [
    "## #3: `time` Table\n",
    "#### Extract Data for Time Table\n",
    "- Filter records by `NextSong` action\n",
    "- Convert the `ts` timestamp column to datetime\n",
    "  - Hint: the current timestamp is in milliseconds\n",
    "- Extract the timestamp, hour, day, week of year, month, year, and weekday from the `ts` column and set `time_data` to a list containing these values in order\n",
    "  - Hint: use pandas' [`dt` attribute](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.dt.html) to access easily datetimelike properties.\n",
    "- Specify labels for these columns and set to `column_labels`\n",
    "- Create a dataframe, `time_df,` containing the time data for this file by combining `column_labels` and `time_data` into a dictionary and converting this into a dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true
   },
   "outputs": [],
   "source": [
    "df = df[df[\"page\"] == \"NextSong\"]\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true
   },
   "outputs": [],
   "source": [
    "t = pd.to_datetime(df[\"ts\"], unit=\"ms\")\n",
    "t.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true
   },
   "outputs": [],
   "source": [
    "#  hour, day, week of year, month, year, and weekday\n",
    "time_data = ([tm.timestamp(), tm.hour, tm.day, tm.week, tm.month, tm.year, tm.weekday()] for tm in t)\n",
    "column_labels = ([\"timestamp\", \"hour\", \"day\", \"week_of_year\", \"month\", \"year\", \"weekday\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true
   },
   "outputs": [],
   "source": [
    "time_df = pd.DataFrame(time_data, columns=column_labels)\n",
    "time_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true
   },
   "source": [
    "#### Insert Records into Time Table\n",
    "Implement the `time_table_insert` query in `sql_queries.py` and run the cell below to insert records for the timestamps in this log file into the `time` table. Remember to run `create_tables.py` before running the cell below to ensure you've created/resetted the `time` table in the sparkify database."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true
   },
   "outputs": [],
   "source": [
    "cur.execute(\"\"\"\n",
    "drop table time\n",
    "\"\"\")\n",
    "conn.commit()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true
   },
   "outputs": [],
   "source": [
    "cur.execute(\"\"\"\n",
    "create table if not exists time(id serial primary key, timestamp varchar, hour int, day int, week_of_year int, month int, year int, weekday int)\n",
    "\"\"\")\n",
    "conn.commit()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true
   },
   "outputs": [],
   "source": [
    "for i, row in time_df.iterrows():\n",
    "    cur.execute(\"\"\"\n",
    "        insert into time(timestamp, hour, day, week_of_year, month, year, weekday) values(%s,%s,%s,%s,%s,%s,%s)\n",
    "    \"\"\", list(row))\n",
    "    conn.commit()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true
   },
   "outputs": [],
   "source": [
    "cur.execute(\"\"\"\n",
    "select * from time limit 5;\n",
    "\"\"\")\n",
    "conn.commit()\n",
    "row = cur.fetchone()\n",
    "while row:\n",
    "   print(row)\n",
    "   row = cur.fetchone()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true
   },
   "source": [
    "Run `test.ipynb` to see if you've successfully added records to this table."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true
   },
   "source": [
    "## #4: `users` Table\n",
    "#### Extract Data for Users Table\n",
    "- Select columns for user ID, first name, last name, gender and level and set to `user_df`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true
   },
   "outputs": [],
   "source": [
    "user_df = df[[\"userId\", \"firstName\", \"lastName\", \"gender\", \"level\"]]\n",
    "user_df.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true
   },
   "source": [
    "#### Insert Records into Users Table\n",
    "Implement the `user_table_insert` query in `sql_queries.py` and run the cell below to insert records for the users in this log file into the `users` table. Remember to run `create_tables.py` before running the cell below to ensure you've created/resetted the `users` table in the sparkify database."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true
   },
   "outputs": [],
   "source": [
    "cur.execute(\n",
    "\"\"\"\n",
    "    drop table users\n",
    "\"\"\")\n",
    "conn.commit()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true
   },
   "outputs": [],
   "source": [
    "cur.execute(\n",
    "\"\"\"\n",
    "    create table if not exists users(user_id int, first_name varchar, last_name varchar, gender varchar, level varchar, primary key(user_id))\n",
    "\"\"\")\n",
    "conn.commit()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true
   },
   "outputs": [],
   "source": [
    "for i, row in user_df.iterrows():\n",
    "    cur.execute(\"\"\"\n",
    "        insert into users(user_id, first_name, last_name, gender, level) values(%s, %s, %s, %s, %s) ON CONFLICT (user_id) DO UPDATE SET level = EXCLUDED.level\n",
    "    \"\"\", row)\n",
    "    conn.commit()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true
   },
   "source": [
    "Run `test.ipynb` to see if you've successfully added records to this table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true
   },
   "outputs": [],
   "source": [
    "cur.execute(\"\"\"\n",
    "select * from users limit 3;\n",
    "\"\"\")\n",
    "conn.commit()\n",
    "row = cur.fetchone()\n",
    "while row:\n",
    "   print(row)\n",
    "   row = cur.fetchone()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true
   },
   "source": [
    "## #5: `songplays` Table\n",
    "#### Extract Data and Songplays Table\n",
    "This one is a little more complicated since information from the songs table, artists table, and original log file are all needed for the `songplays` table. Since the log file does not specify an ID for either the song or the artist, you'll need to get the song ID and artist ID by querying the songs and artists tables to find matches based on song title, artist name, and song duration time.\n",
    "- Implement the `song_select` query in `sql_queries.py` to find the song ID and artist ID based on the title, artist name, and duration of a song.\n",
    "- Select the timestamp, user ID, level, song ID, artist ID, session ID, location, and user agent and set to `songplay_data`\n",
    "\n",
    "#### Insert Records into Songplays Table\n",
    "- Implement the `songplay_table_insert` query and run the cell below to insert records for the songplay actions in this log file into the `songplays` table. Remember to run `create_tables.py` before running the cell below to ensure you've created/resetted the `songplays` table in the sparkify database."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true
   },
   "outputs": [],
   "source": [
    "cur.execute(\n",
    "\"\"\"\n",
    "    drop table songplays\n",
    "\"\"\")\n",
    "conn.commit()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true
   },
   "outputs": [],
   "source": [
    "cur.execute(\n",
    "\"\"\"\n",
    "create table if not exists songplays(start_time varchar not null, user_id int not null, level varchar, song_id varchar, artist_id varchar, session_id int, location varchar, user_agent varchar)\n",
    "\"\"\")\n",
    "conn.commit()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true
   },
   "outputs": [],
   "source": [
    "for index, row in df.iterrows():\n",
    "\n",
    "    # get songid and artistid from song and artist tables\n",
    "    cur.execute(\"\"\"\n",
    "        select s.song_id, a.artist_id\n",
    "        from songs as s\n",
    "        join artists as a on a.artist_id = s.artist_id\n",
    "        where s.title = %s and a.name = %s and s.duration = %s;\n",
    "    \"\"\", (row.song, row.artist, row.length))\n",
    "    results = cur.fetchone()\n",
    "    \n",
    "    if results:\n",
    "        songid, artistid = results\n",
    "    else:\n",
    "        songid, artistid = None, None\n",
    "\n",
    "    # insert songplay record\n",
    "    songplay_data = (row.ts, row.userId, row.level, songid, artistid, row.sessionId, row.location, row.userAgent)\n",
    "    cur.execute(\"\"\"\n",
    "        insert into songplays(start_time, user_id, level, song_id, artist_id, session_id, location, user_agent) values(%s, %s, %s, %s, %s, %s, %s, %s)\n",
    "    \"\"\", songplay_data)\n",
    "    conn.commit()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true
   },
   "outputs": [],
   "source": [
    "curs = conn.cursor()\n",
    "curs.execute(\"ROLLBACK\")\n",
    "conn.commit()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true
   },
   "outputs": [],
   "source": [
    "cur.execute(\"\"\"\n",
    "select * from songplays limit 3;\n",
    "\"\"\")\n",
    "conn.commit()\n",
    "row = cur.fetchone()\n",
    "while row:\n",
    "   print(row)\n",
    "   row = cur.fetchone()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true
   },
   "source": [
    "Run `test.ipynb` to see if you've successfully added records to this table."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true
   },
   "source": [
    "# Close Connection to Sparkify Database"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "editable": true
   },
   "outputs": [],
   "source": [
    "conn.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "editable": true
   },
   "source": [
    "# Implement `etl.py`\n",
    "Use what you've completed in this notebook to implement `etl.py`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "drop table users;\n",
      "failed  drop table users;\n",
      "drop table songs;\n",
      "failed  drop table songs;\n",
      "drop table artists;\n",
      "failed  drop table artists;\n",
      "drop table time;\n",
      "failed  drop table time;\n",
      "drop table songplays;\n",
      "failed  drop table songplays;\n",
      "\n",
      "    create table if not exists users(\n",
      "        user_id int, \n",
      "        first_name varchar, \n",
      "        last_name varchar, \n",
      "        gender varchar, \n",
      "        level varchar, \n",
      "        primary key(user_id)\n",
      "    );\n",
      "\n",
      "\n",
      "    create table if not exists songs(\n",
      "        song_id varchar, \n",
      "        title varchar not null, \n",
      "        artist_id varchar, \n",
      "        year int, \n",
      "        duration decimal not null, \n",
      "        primary key (song_id)\n",
      "    );\n",
      "\n",
      "\n",
      "    create table if not exists artists(\n",
      "        artist_id varchar, \n",
      "        name varchar not null, \n",
      "        location varchar, \n",
      "        latitude float, \n",
      "        longitude float, \n",
      "        primary key (artist_id)\n",
      "    );\n",
      "\n",
      "\n",
      "    create table if not exists time(\n",
      "        start_time bigint, \n",
      "        hour int, \n",
      "        day int, \n",
      "        week int, \n",
      "        month int, \n",
      "        year int, \n",
      "        weekday int\n",
      "    );\n",
      "\n",
      "\n",
      "    create table if not exists songplays(\n",
      "        songplay_id serial primary key,\n",
      "        start_time bigint not null, \n",
      "        user_id int not null, \n",
      "        level varchar, \n",
      "        song_id varchar, \n",
      "        artist_id varchar, \n",
      "        session_id int, \n",
      "        location varchar, \n",
      "        user_agent varchar,\n",
      "        constraint fk_user foreign key (user_id) references users(user_id),\n",
      "        constraint fk_song foreign key (song_id) references songs(song_id),\n",
      "        constraint fk_artist foreign key (artist_id) references artists(artist_id)\n",
      "    );\n",
      "\n"
     ]
    }
   ],
   "source": [
    "%run -i 'create_tables.py'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "71 files found in data/song_data\n",
      "1/71 files processed.\n",
      "2/71 files processed.\n",
      "3/71 files processed.\n",
      "4/71 files processed.\n",
      "5/71 files processed.\n",
      "6/71 files processed.\n",
      "7/71 files processed.\n",
      "8/71 files processed.\n",
      "9/71 files processed.\n",
      "10/71 files processed.\n",
      "11/71 files processed.\n",
      "12/71 files processed.\n",
      "13/71 files processed.\n",
      "14/71 files processed.\n",
      "15/71 files processed.\n",
      "16/71 files processed.\n",
      "17/71 files processed.\n",
      "18/71 files processed.\n",
      "19/71 files processed.\n",
      "20/71 files processed.\n",
      "21/71 files processed.\n",
      "22/71 files processed.\n",
      "23/71 files processed.\n",
      "24/71 files processed.\n",
      "25/71 files processed.\n",
      "26/71 files processed.\n",
      "27/71 files processed.\n",
      "28/71 files processed.\n",
      "29/71 files processed.\n",
      "30/71 files processed.\n",
      "31/71 files processed.\n",
      "32/71 files processed.\n",
      "33/71 files processed.\n",
      "34/71 files processed.\n",
      "35/71 files processed.\n",
      "36/71 files processed.\n",
      "37/71 files processed.\n",
      "38/71 files processed.\n",
      "39/71 files processed.\n",
      "40/71 files processed.\n",
      "41/71 files processed.\n",
      "42/71 files processed.\n",
      "43/71 files processed.\n",
      "44/71 files processed.\n",
      "45/71 files processed.\n",
      "46/71 files processed.\n",
      "47/71 files processed.\n",
      "48/71 files processed.\n",
      "49/71 files processed.\n",
      "50/71 files processed.\n",
      "51/71 files processed.\n",
      "52/71 files processed.\n",
      "53/71 files processed.\n",
      "54/71 files processed.\n",
      "55/71 files processed.\n",
      "56/71 files processed.\n",
      "57/71 files processed.\n",
      "58/71 files processed.\n",
      "59/71 files processed.\n",
      "60/71 files processed.\n",
      "61/71 files processed.\n",
      "62/71 files processed.\n",
      "63/71 files processed.\n",
      "64/71 files processed.\n",
      "65/71 files processed.\n",
      "66/71 files processed.\n",
      "67/71 files processed.\n",
      "68/71 files processed.\n",
      "69/71 files processed.\n",
      "70/71 files processed.\n",
      "71/71 files processed.\n",
      "30 files found in data/log_data\n",
      "1/30 files processed.\n",
      "2/30 files processed.\n",
      "3/30 files processed.\n",
      "4/30 files processed.\n",
      "5/30 files processed.\n",
      "6/30 files processed.\n",
      "7/30 files processed.\n",
      "8/30 files processed.\n",
      "9/30 files processed.\n",
      "10/30 files processed.\n",
      "11/30 files processed.\n",
      "12/30 files processed.\n",
      "13/30 files processed.\n",
      "14/30 files processed.\n",
      "15/30 files processed.\n",
      "16/30 files processed.\n",
      "17/30 files processed.\n",
      "18/30 files processed.\n",
      "19/30 files processed.\n",
      "20/30 files processed.\n",
      "21/30 files processed.\n",
      "22/30 files processed.\n",
      "23/30 files processed.\n",
      "24/30 files processed.\n",
      "25/30 files processed.\n",
      "26/30 files processed.\n",
      "27/30 files processed.\n",
      "28/30 files processed.\n",
      "29/30 files processed.\n",
      "30/30 files processed.\n"
     ]
    }
   ],
   "source": [
    "%run -i 'etl.py'"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}