Out-of-Core hybrid Apache Arrow/NumPy DataFrame for Python, ML, visualization and exploration of big tabular data at a billion rows per second 🚀

the portable Python dataframe library

Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.

Work with bioinformatic files using Arrow, Polars, and/or DuckDB

Command-line interface to quickly generate fake CSV and JSON data

Exploring Chicago crimes dataset with Jupyter notebooks, DuckDB, Malloy and new Panel/PyScript data and dashboard tools.

Converts a whole subdirectory with a big (or small) volume of PDF documents to a dataset (pandas DataFrame) with error tracking and choice of features

(PoC) A very memory-efficient way to read data from PostgreSQL

db2ixf is a python package with a CLI that simplifies the parsing and processing of IBM Integration eXchange Format (IXF) files.

A web application for viewing Apache Parquet files . This is a Python + Flask application

Reading both XLSX and XLSB files, fast and memory-safe, with Python, into PyArrow

Poor mans simple python api for creating a local or remote datalake based on several (pyarrow) datasets using duckdb

highspeed timeseries pandas dataframe database

Dremio Arrow Flight Client

ibm_db extension to load a pyarrow table to db2

Concise interface to cache numpy arrays and pandas dataframes

Python scripts to process, and analyze log files using PySpark.

Seamlessly switch Pandas DataFrame backend to PyArrow.

Python scripts to download, process, and analyze NYC TLC trip data

A small cast tookit class drived from _ParquetDatasetV2 to support cast in filters argument