✨ A Pydantic to PySpark schema library

An open source, standard data file format for graph data storage and retrieval.

the portable Python dataframe library

Pandas, Sci-kit, SparkML

Dataframe Performance Comparison - Polars, Pandas on Spark, and Pandas

Jupyter magics and kernels for working with remote Spark clusters

Simple and Distributed Machine Learning

💜🌈📊 A Data Engineering Project that implements an ETL data pipeline using Dagster, Apache Spark, Streamlit, MinIO, Apache Superset, Dbt 🌺

Example of local pyspark setup including DeltaLake for unit-testing

Hopsworks - Data-Intensive AI platform with a Feature Store

PySpark script to aggregate small parquet files in a prefix into larger files. Designed to be run on AWS Glue

PySpark code samples designed for AWS Glue

🌈📊📈 The Zillow Home Value Prediction project employs linear regression models on Kaggle datasets to forecast house prices. 📉💰Using Apache Spark (PySpark) within a Docker setup enables efficient data preprocessing, exploration, analysis, visualization, and model building with distributed computing for parallel computation.

Apache Linkis builds a computation middleware layer to facilitate connection, governance and orchestration between the upper applications and the underlying data engines.

Large dataSet of IPL Data till 2017 analysis using PySpark.

Generate relevant synthetic data quickly for your projects. The Databricks Labs synthetic data generator (aka `dbldatagen`) may be used to generate large simulated / synthetic data sets for test, POCs, and other uses in Databricks environments including in Delta Live Tables pipelines

Big Data Modeling, MapReduce, Spark, PySpark @ Santa Clara University

Apache Spark Connector for Azure Cosmos DB

This repo contains Big Data Project, its about "Real Time Twitter Sentiment Analysis via Kafka, Spark Streaming, MongoDB and Django Dashboard".

State of the Art Natural Language Processing