This is a project based on Udacity data engineering project tutorial:
In this project, we apply Data Modeling with Postgres and build an ETL pipeline using Python. A startup wants to analyze the data they've been collecting on songs and user activity on their new music streaming app. Currently, they are collecting data in json format and the analytics team is particularly interested in understanding what songs users are listening to.
- Modifiy the pandas dataframe and use to_sql method to insert all records at once instead of inserting row by row
- Add other sql queries types
- test.ipynb displays the first few rows of each table to let you check your database.
- create_tables.py drops and creates your tables. You run this file to reset your tables before each time you run your ETL scripts.
- etl.ipynb reads and processes a single file from song_data and log_data and loads the data into your tables. This notebook contains detailed instructions on the ETL process for each of the tables.
- etl.py reads and processes files from song_data and log_data and loads them into your tables. You can fill this out based on your work in the ETL notebook.
- sql_queries.py contains all your sql queries, and is imported into the last three files above. README.md provides discussion on your project.
- First run Python create_table.py to create relevant table
- Use test.ipynb to test if records have been inserted
- Run etl.ipynb to insert records