Migrate an Existing dbt Project

This guide will describe how to bring existing dbt code into the your Meltano project. Meltano uses some suggested patterns for organizing your dbt project so it integrates well with core features of Meltano like environments. You can organize your project whatever way you chose but this guide will describe how to import it so it matches the default transformer installation.

Pre-requisites #

As always, we highly recommend git versioning your Meltano project prior to following this guide so you have the ability to roll back and not affect your existing Meltano project configurations.

Add dbt Transformer #

Add your adapter-specific dbt variant (e.g. dbt-postgres) that can be found on MeltanoHub.

meltano add transformer dbt-<adapter_name>

# For example
meltano add transformer dbt-postgres

Next configure your transformer to include database names, connection credentials, etc. See the transform data guide for more details. Or use the interactive config flag to follow prompts.

meltano config dbt-snowflake set --interactive

Once you’ve configured your transformer you should be able to run the following command to test your connection and credentials.

meltano invoke dbt-postgres debug

Migrating dbt Code Into Meltano #

Note that adding a transformer creates scaffolding within your Meltano /transform directory including a dbt_project.yml and /profile/profiles.yml. If you have an existing dbt project you will already have your own version of these files in your other repo so we’ll describe how to merge what you have and what Meltano provides and expects.

Meltano’s Default Structure For dbt #

Meltano expects dbt project files to exist in the default directories listed below. You can either place your files in the appropriate directories or you can update the given dbt_project.yml to follow the directory structure of your existing project, if thats preferred.

data - this is where seed files are stored. See seeds dbt docs
models - this is where models are stored. See models dbt docs
analysis - analysis sql that shouldnt be materialized. See analyses dbt docs
tests - this is where singular dbt test are stored. See tests dbt docs
macros - jinja macros. See macros dbt docs
snapshots - this is where snapshot models are stored. See snapshots dbt docs

dbt Profiles #

Meltano’s default dbt project scaffolding comes with a profiles.yml (see dbt profiles docs for details) that is configured to take advantage of the environments feature. This means that you configure dbt using the provided Meltano settings and they get automatically passed to dbt based on what Meltano environment is active. Meltano’s dbt installation comes with pre-configured dbt targets mapped to the default environment names (i.e. dev, staging, prod), avoiding the need to toggle credentials manually and allowing sharing of settings/credentials across plugins.

Custom `dbt_projects.yml` Configurations #

If you had any configurations in your dbt_project.yml such as definitions of how models are materialized, target databases, schemas, etc. you can directly copy them into your new Meltano dbt_project.yml file.

Again Meltano doesn’t require this structure, any valid dbt project will work, but this is the default recommended structure with some base configurations for a simple integration between Meltano and dbt.

Edit this page!