datavault4dbt by Scalefree International GmbH
- Staging Area (For Hashing, prejoins and ghost records)
- Hubs, Links & Satellites (allowing multiple deltas)
- Non-Historized Links and Satellites
- Multi-Active Satellites
- Virtualized End-Dating (in Satellites)
- Reference Hubs, - Satellites, and - Tables
- PIT Tables
- Hook for Cleaning up PITs
- Snapshot Control
With datavault4dbt you will get a lot of awesome features, including:
- A Data Vault 2.0 implementation congruent to the original Data Vault 2.0 definition by Dan Linstedt
- Ready for both Persistent Staging Areas and Transient Staging Areas, due to the allowance of multiple deltas in all macros, without loosing any intermediate changes- Enforcing standards in naming conventions by implementing global variables for technical columns
- A fully auditable solution for a Data Warehouse
- Creating a centralized, snapshot-based Business interface by using a centralized snapshot table supporting logarithmic logic
- A modern insert-only approach that avoids updating data
- Optimizing incremental loads by implementing a high-water-mark that also works for entities that are loaded from multiple sources
- A straight-forward, standardized approach to conduct agile datawarehouse development cycles
To use the macros efficiently, there are a few prerequisites you need to provide:
- Flat & Wide source data, available within your target database
- Load Date column that represents the arriving time in the source data storage
- Record Source column that gives information about where the source data is coming from (e.g. the file location inside a Data Lake)
- Learn more about dbt in the docs
- Check out Discourse for commonly asked questions and answers
- Join the chat on Slack for live discussions and support
- Check out the blog for the latest news on dbt's development and best practices
- Find dbt events near you
- Check out the Scalefree-Blog
Currently supported platforms are:
- Google Bigquery
- Exasol
- Snowflake
- PostgreSQL
- Amazon Redshift
- Microsoft Azure Synapse
We are working continuously at high pressure to adapt the package for large variety of different platforms. In the future, the package will hopefully be available for SQL Server, Oracle and many more.
- Include this package in your
packages.yml-- check here for installation instructions. - Run
dbt deps
For further information on how to install packages in dbt, please visit the following link: https://docs.getdbt.com/docs/building-a-dbt-project/package-management
datavault4dbt is highly customizable by using many global variables. Since they are applied on multiple levels, a high rate of standardization across your data vault 2.0 solution is guaranteed. The default values of those variables are set inside the packages dbt_project.yml and should be copied to your own dbt_project.yml. For an explanation of all global variables see the wiki.
The datavault4dbt package provides macros for Staging and Creation of all DataVault-Entities you need, to build your own DataVault2.0 solution. The usage of the macros is well-explained in the documentation: https://github.com/ScalefreeCOM/datavault4dbt/wiki
View our contribution guidelines
For questions, feedback, etc. reach out to us via datavault4dbt@scalefree.com!