Sql Migration

A pipeline for transforming Extracting, Loading and Transforming data to businesses “The only constant thing in life is change”. ~Heraclitus Or more radically: “When you are finished changing, you are finished.” ~Benjamin Franklin With the current speed of new innovations in the data space, building modular, loosely coupled, and programmatically adaptable solutions for your tech stack is essential to move fast and deliver value to your stakeholders. In this case, your AI startup has gotten some sales, some revenue, and some publicity = traction. This has led to you getting an investor, with some tech background. Your investor is insisting that a new, more scalable, tech-stack be deployed as a condition for the 2nd tranche of her investment into your firm.

Sql Migration Project

A fully dockerized ELT pipeline project, using Microsoft SQL, dbt, Apache Airflow, and Apache Superset.
Explore the docs »

Table of Contents

1. About The Project
   • Built With
2. Getting Started
   • Prerequisites
   • Installation
3. Usage
4. Roadmap
5. Contributing
6. License
7. Contact
8. Acknowledgements

Using a docker-compose file, developed a completely dockerized ELT pipeline with Microsft SQL for data storage, Airflow for automation and orchestration, DBT for data transformation, and a Redash dashboard connected to the PostgreSQL database.

Built With

Tech Stack used in this project

• MSSql
• Apache Airflow
• dbt
• Apache Superset

Getting Started

Prerequisites

Make sure you have docker installed on local machine.

• Docker
• DockerCompose

Installation

1. Clone the repo

   git clone https://github.com/tutorialcreation/sqlmigration.git

2. Run

    docker-compose build
    docker-compose up

3. Open Airflow web browser

   Navigate to `http://localhost:8000/` on the browser
   activate and trigger dbt_load_dag
   activate and trigger dbt_dbt_dag

4. Access redash dashboard

   Navigate to `http://localhost:5000/` on the browser

5. Access your PostgreSQL database using adminar

   Navigate to `http://localhost:8080/` on the browser
   choose PostgreSQL databse
   use `root` for username
   use `pssd` for password

Contributing

Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
3. Commit your Changes (git commit -m 'Add some AmazingFeature')
4. Push to the Branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

License

Distributed under the MIT License. See LICENSE for more information.

Contact

Martin Luther - @email - tutorialcreation81@gmail.com

Acknowledgements

• 10 Academy