Optimizing Dataframes and Processing in Chunks

In this project we will explore two very important strategies for handling large datasets and dealing with limited hardware resources:

• Optimizing Memory Footprint
• Processing the Data in Chunks

As we will see, the combination of these two powerful approaches will significantly improve our access to the data, independently of its size.

Head over to Optimizing Dataframes and Processing in Chunks.ipynb to start reading.

Requirements

For this project we implemented the following tools:

• Python
• Pandas