MapReduce, Spark, Java, and Scala for Data Algorithms Book

Big Data Modeling, MapReduce, Spark, PySpark @ Santa Clara University

HADOOP 3.1.0 winutils

hadoop-cos（CosN文件系统）为Apache Hadoop、Spark以及Tez等大数据计算框架集成提供支持，可以像访问HDFS一样读写存储在腾讯云COS上的数据。同时也支持作为Druid等查询与分析引擎的Deep Storage

Export Hadoop YARN (resource-manager) metrics in prometheus format

Containerized Apache Hive Metastore for horizontally scalable Hive Metastore deployments

A fast, scalable and distributed community detection algorithm based on CEIL scoring function.

A Spark application to merge small files on Hadoop

The implementation of Apache Spark (combine with PySpark, Jupyter Notebook) on top of Hadoop cluster using Docker

Implement a Hive data warehouse to store meaningful data, apply Machine Learning like Clustering or Regression for dealing with business problems

Some simple, kinda introductory projects based on Apache Hadoop to be used as guides in order to make the MapReduce model look less weird or boring.

The source code developed and used for the purposes of my thesis with the same title under the guidance of my supervisor professor Vasilis Mamalis for the Department of Informatics and Computer Engineering of the University of West Attica.

An email spam filter using Apache Spark’s ML library

Samples related to data engineering, e.g. spark, embulk, airflow, etc.

Links

A BASH script to setup Apache Hadoop and Apache Hive with Derby database on Debian GNU/Linux

Apache Hadoop. Apache Hadoop is a collection of open-source software utilities that facilitate using a network of many computers to solve problems involving massive amounts of data and computation. It provides a software framework for distributed storage and processing of big data using the MapReduce programming model. Originally designed for co…

This project implemented a lambda architecture for analyzing domestic flight data in the US from 2009 to 2020. It used Apache Spark for batch processing, Spark Streaming for real-time analysis, and SVM models to predict flight cancellations and delays, with Docker for cluster management and Grafana for real-time visualization.

Built a Large Scale Distributed Data Processing system for Streaming Analytics using Hadoop Ecosystem (Apache Spark and HDFS), in Cloud for real-time spatial analytics.