Prefect is a workflow orchestration tool empowering developers to build, observe, and react to data pipelines
-
Updated
Jun 11, 2024 - Python
Prefect is a workflow orchestration tool empowering developers to build, observe, and react to data pipelines
Dagster Labs' open-source data platform, built with Dagster.
Enterprise-grade, production-hardened, serverless data lake on AWS
An orchestration platform for the development, production, and observation of data assets.
data load tool (dlt) is an open source Python library that makes data loading easy 🛠️
The open source high performance ELT framework powered by Apache Arrow
The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
Apache Superset is a Data Visualization and Data Exploration Platform
pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).
Apache DevLake is an open-source dev data platform to ingest, analyze, and visualize the fragmented data from DevOps tools, extracting insights for engineering excellence, developer experience, and community growth.
The code to follow along our tutorials for the dlt rest_api source
Infinitely scalable, event-driven, language-agnostic orchestration and scheduling platform to manage millions of workflows declaratively in code.
CSVs sliced, diced & analyzed.
fluent iteration
🧙 Build, run, and manage data pipelines for integrating and transforming data.
Yet another redundant workflow engine
Turns Data and AI algorithms into production-ready web applications in no time.
Developer-first embedded analytics
Add a description, image, and links to the data-engineering topic page so that developers can more easily learn about it.
To associate your repository with the data-engineering topic, visit your repo's landing page and select "manage topics."