Mirror of Apache Beam Site

GCP Dataflow pipeline with BigQuery as source and side input

Collection of Kotlin tricks

Serverless data ingest pipeline on Google Cloud Platform

Pipeline para ingestão e tratamento de dados utilizando o Apache Beam

Desenvolvimento de um pipeline de dados utilizando Apache Beam para orquestrar o fluxo e Python para capturar e tratar os dados. Com os dados já refinados, foram utilizadas as bibliotecas Pandas e Matplotlib para desenvolver uma análise exploratória dos dados.

Implementing K-means clustering in sequential, streaming, and distributed formats using Apache Beam.

This repo contains step by step guide on how I executed this challenge. I won't disclaim challenge information, as this repo was made specifically for the team that created the test.

This is a data scraping project that sources data from the Houzz e-commerce platform, the CNN YouTube channel, and the TedTalk official website. The implementation uses the Apache Beam framework to build an ETL pipeline and write the results into an Elasticsearch database. The final step visualizes the crawler results using Kibana.

🍦 Serve doddle-model in a pipeline implemented with Apache Beam

GCP Dataflow pipeline with mapreduce in python

BigData knowledge system（大数据知识体系）.

Weekly Assignments

Adtech Logs processing Pipeline with Apache Beam, Cloud Dataflow, Java, Protocol Buffer. | Data Analysis with BigQuery

A data pipeline to ingest, process, store storm events datasets so we can access them through different means.

This video present a real world use case developed with Apache Beam Java and launched with the serverless Dataflow runner in Google Cloud Platform. The job read a Json file from Cloud Storage, applies some transformations and write the result to a BigQuery table.

The scripts in this repo will build the Apache Beam Java SDK packages, using Cloud Build and Artifact Registry, for a personal Beam fork.

Efficient Python data pipeline leveraging Apache Beam and Google Cloud Dataflow to update a Bucket with data concerning daily prices of instruments extracted from BMF website, serving as input for other data pipelines. The code generates a dataflow template, which is then scheduled to run periodically using Cloud Scheduler + Cloud Functions.

Mirror of Apache Beam

Evaluating Apache Beam batch, streaming, SQL, etc