Twitter Data Pipeline with apache-airflow & MinIO

Table of Contents

• About
• Getting Started
• Usage
• Resources
• etc.

About

A simple example of Data Pipeline using apache-airflow (Orchestrator) and MinIO(Object Storage like s3).

Getting Started

Below is a design of the project. (The drawio file can be found in docs/architecture.drawio:

Prerequisites

• docker-compose
• How to generate Twitter API Token

Usage

1. We need to create and .env from sample.env:

cp sample.env .env

2. Add the Twitter Bearer Token in the .env file as below:

TWITTER_BEARER_TOKEN="vNVxBVjj-0yhF!Ipc-p7Nrzl7C2wISOI6BLXVk087/jJS4auIp0SKSXI/7npGy1kl7xDXxRuJ55Lor5FHI!6!!a5v0!IrxCDYQDEgMBQzOZivgIEpQJsvC4A0nqFbqxA"

3. We can simply run the pipeline using docker-compose.

To start

docker compose up -d

To shutdown

docker compose down

4. Then we can connect the below respectively:

• Apache-Airflow: http://localhost:8080
• MinIO Console: http://localhost:9090

Resources

• How to generate Twitter API Token
• apache-airflow
• MinIO
• docker-compose

License

This project is licensed under the MIT License - see the LICENSE file for details.

Credits

See as you fit.

Contact

If you have any questions or would like to get in touch, you can email: mike.kenneth47@gmail.com OR twitter