Upserts, Deletes And Incremental Processing on Big Data.
This is a repo with links to everything you'd ever want to learn about data engineering
FLaNK AI Weekly covering Apache NiFi, Apache Flink, Apache Kafka, Apache Spark, Apache Iceberg, Apache Ozone, Apache Pulsar, and more...
Code for blog at: https://www.startdataengineering.com/post/docker-for-de/
Ce dépôt GitHub contient un document détaillé sur les bases du langage Scala.
An end-to-end data engineering pipeline that orchestrates data ingestion, processing, and storage using Apache Airflow, Python, Apache Kafka, Apache Zookeeper, Apache Spark, and Cassandra. All components are containerized with Docker for easy deployment and scalability.
Run your first analysis project on Apache Zeppelin using Scala (Spark), Shell, and SQL
SparkSQL.jl enables Julia programs to work with Apache Spark data using just SQL.
This repository will help you to learn about databricks concept with the help of examples. It will include all the important topics which we need in our real life experience as a data engineer. We will be using pyspark & sparksql for the development. At the end of the course we also cover few case studies.
Developed a real-time streaming analytics pipeline using Apache Spark to calculate and store KPIs for e-commerce sales data, including total volume of sales, orders per minute, rate of return, and average transaction size. Used Spark Streaming to read data from Kafka, Spark SQL to calculate KPIs, and Spark DataFrame to write KPIs to JSON files.
PySpark es una biblioteca de procesamiento de datos distribuidos en Python que permite procesar grandes volúmenes de datos en clústeres utilizando el framework Apache Spark, ofreciendo un alto rendimiento y un conjunto de herramientas integradas para el análisis y manejo de datos a gran escala.
US superstore opening analysis
a brief analysis to the most common words in Dracula, by Bram Stoker
A Capstone Project that covers several aspects of Data Engineering (Data Exploration, Cleaning, Modeling, Pipelining, Processing)
You will find here the demo codes for my Data+AI 2020 talk about customizing Apache Spark state store.
Repository for Lab “Distributed Big Data Analytics” (MA-INF 4223), University of Bonn
Use this project to join data from multiple csv files. Currently in this project we support one to one and one to many join. Along with this you can find how to use kafka producer efficiently with spark.
Add a description, image, and links to the apachespark topic page so that developers can more easily learn about it.
To associate your repository with the apachespark topic, visit your repo's landing page and select "manage topics."