automate-dv is a free, open source data engineering & integration project written in several languages and released under Apache-2.0. It has 598 GitHub stars, 157 forks and 53 open issues, and was last pushed 9 months ago. On this registry it ranks #49 of 53 tracked projects in Data Engineering & Integration, with 5 head-to-head comparisons available. It gained 2 stars over the last 30 tracked days.

What is automate-dv?

AutomateDV is a free, Apache-2.0-licensed dbt package that generates and executes the ETL needed to build and load a Data Vault 2.0 compliant data warehouse on the data platform a team already runs.

What it is

AutomateDV is a package for dbt, the open source data engineering tool, and it is installed and run inside an existing dbt project rather than as a standalone server. It is distributed through the dbt Hub as Datavault-UK/automate_dv and is maintained by Datavault. The package is compatible with dbt versions greater than or equal to 1.9.x and less than or equal to 3.0.0, and it is also listed as compatible with dbt Fusion. It targets data platforms recorded in a platform support matrix, with Snowflake appearing among the project's topics alongside data lake, data warehouse, and ELT and ETL concerns.

The concrete problem it solves is the volume of repeated hand-written SQL needed to load Data Vault 2.0 structures. Instead of writing each hub, link, and satellite load by hand, a data modeller supplies mapping metadata and the package generates and executes the ETL from it. The intended division of labour is explicit: the modeller produces most of the ETL code directly from mapping metadata, and ETL developers concentrate on the estimated 5% of SQL that differs from the standard pattern. That replaces routine Data Vault boilerplate with templated macro calls, and it reduces the error surface that comes with maintaining that code manually.

Key capabilities

  • Generates and executes Data Vault 2.0 ETL from dbt models, driven by metadata rather than hand-written load SQL.
  • Exposes template macros called directly from a model, for example {{ automate_dv.hub(src_pk, src_nk, src_ldts, src_source, source_model) }}.
  • Uses a documented three-step usage pattern: a config(...) block, metadata variables such as src_pk, and a macro call per table.
  • Runs the generated SQL in a multi-threaded fashion through dbt.
  • Produces documentation and data flow diagrams generated by dbt alongside the models.
  • Ships a worked example project in the companion repository automate-dv-demo, with documentation under the Worked Example section.
  • Is published on the dbt Hub with a maintained platform support matrix covering the supported data platforms.

Who uses it and how

  • Data modellers who hold the mapping metadata and want to generate most of the Data Vault ETL without writing it by hand.
  • ETL developers on the same team who take the remaining SQL that differs from the generated pattern.
  • dbt teams building Data Vault 2.0 warehouses on Snowflake and on other platforms listed in the platform support matrix.
  • Organisations loading a data lake or data warehouse with ELT pipelines that need hub, link, and satellite loads produced consistently.
  • New adopters who follow the worked example project and documentation, and who get support through the project's Slack channel.

Getting started

Install the package from the dbt Hub as Datavault-UK/automate_dv, following the installation instructions there or the general dbt guidance on installing packages. Then create a model for the table, provide the metadata, and call the appropriate template macro.

How it compares

The facts provided do not name any paid products that AutomateDV replaces, and no competing tool is named either, so on the evidence given it stands alone in this registry. Comparisons that can be made are against hand-written Data Vault ETL inside a dbt project, which is the practice it is built to displace. Any broader claim about alternatives would go beyond what the supplied facts support.

When to use it — and when not to

A self-hoster must already operate a dbt project and a data platform that appears in the platform support matrix, together with whatever scheduling and warehouse infrastructure dbt runs against. Teams with no dbt investment, or teams not committed to the Data Vault 2.0 modelling approach, should not pick it, because the package assumes both. The README is thin and delegates most operational detail to external documentation on Read the Docs, so a prospective adopter should expect to read the platform support matrix and the worked example before committing.

project readme (upstream, from github) — read inline

AutomateDV

Documentation Status Slack

dbt Versions

dbt Fusion

Changelog and past doc versions

AutomateDV by Datavault

Build your own Data Vault data warehouse! AutomateDV is a free to use dbt package that generates & executes the ETL you need to run a Data Vault 2.0 Data Warehouse on your data platform.

What does AutomateDV offer?

  • productivity gains, fewer errors
  • multi-threaded execution of the generated SQL
  • your data modeller can generate most of the ETL code directly from their mapping metadata
  • your ETL developers can focus on the 5% of the SQL code that is different
  • dbt generates documentation and data flow diagrams

powered by dbt, a registered trademark of dbt Labs

Worked example project

Learn quickly with our worked example:

Supported platforms:

Platform support matrix

Installation

Check dbt Hub for the latest installation instructions, or read the docs for more information on installing packages.

Documentation

ReadTheDocs

Usage

  1. Create a model for your table.
  2. Provide metadata
  3. Call the appropriate template macro
# Configure model
{{- config(...)                          -}}

# Provide metadata
{%- set src_pk = ...                     -%}
...

# Call the macro
{{ automate_dv.hub(src_pk, src_nk, src_ldts,
                   src_source, source_model) }}

Join our Slack Channel

Talk to our developers and other members of our growing community, get support and discuss anything related to AutomateDV or Data Vault 2.0

Join our Slack

Social

LinkedIn Youtube

Awards

innovation awards

Contributing

View our contribution guidelines

License

Apache 2.0

Frequently asked questions

Is automate-dv free to use?

automate-dv is open source under the Apache-2.0 licence. There is no licence fee and no seat count — you can self-host it or, where the project offers one, pay a vendor for a managed version instead.

What does automate-dv do?

A free to use dbt package for creating and loading Data Vault 2.0 compliant Data Warehouses (powered by dbt, an open source data engineering tool, registered t

What is automate-dv written in?

automate-dv's source is publicly available at https://github.com/Datavault-UK/automate-dv, with 598 GitHub stars.