Meltano 2.0 Migration Guide

Note: This document is still a work in progress. Expect further changes, coming soon.

The following list includes all recommended migration tasks as well as breaking changes in Meltano version 2.0.

Recommended #

Migrate to an Adapter-Specific dbt Transformer #

If you previously used the dbt Transformer, we recommend migrating to an adapter-specific installation as per the dbt available adapters documentation.

Install dbt #

This is easy to do! Following the instructions from above to discover and install your chosen adapter:

# list available transformer plugins
meltano discover transformers

# install adapter-specific dbt, e.g. for snowflake
meltano add transformer dbt-snowflake

Update your dbt_project.yml #

Installation of a new Transformer will introduce two important files to your transform/ directory:

• A new profiles.yml file in transform/profiles/<adapter name>/profiles.yml
• A new dbt_project.yml file in transform/dbt_project (<adapter name>).yml

The new profiles.yml will only be used by adapter-specific dbt executions (e.g. dbt-snowflake), and can be customized to meet your requirements. Your existing profiles.yml will remain in use by your existing dbt Transformer plugin (via elt and invoke).

It is likely that the new dbt_project (<adapter name>).yml will contain changes from your previous dbt_project.yml file, especially if you haven’t already upgraded to dbt v1.0. To complete your migration, consolidate dbt_project.yml and dbt_project (<adapter name>).yml into a single file called dbt_project.yml. As this project file will be used by both dbt and dbt-<adapter> Transformer plugins by default, you must ensure you are running an up-to-date installation of plugin dbt if you intend to use both adapter-specific and legacy dbt installs together (not recommended).

If you make use of Transform plugins, these will continue to work as regular dbt packages. However adding new Transform plugins will currently (tracking at #3304) re-add the legacy dbt Transformer plugin. To avoid this we recommend adding Transforms as regular packages directly via dbt as per the dbt Packages documentation.

Remove the dbt Transformer plugin and associated files #

To remove the legacy dbt Transformer plugin, run:

# remove the transformer `dbt`
meltano remove transformer dbt

# remove the file bundle `dbt`
meltano remove files dbt

Removing a file bundle does not remove any files from your transform/ directory. Manually remove transform/profile/profiles.yml to complete clean-up (as adapter-specific installs come with their own profiles.yml in transform/profiles/<adapter name>/profiles.yml).

Removed #

model and dashboard plugin types #

These plugin types provided very basic BI capabilities using Meltano UI. However there are already great 3rd party open source BI solutions in this space, such as the newly added Superset plugin. Meltano model and dashboard plugins have been removed in favour of existing and future 3rd party tools for the same purpose.

transform support in meltano elt #

Meltano 2.0 continues to support extract-load (EL) operations with meltano elt. However, for EL+T operations which also need to transform data, please use meltano run.

transform support in Meltano schedules #

Meltano 2.0 continues to support extract-load (EL) operations in schedules. However, for EL+T operations which also need to transform data, please use the new meltano job add command to create a job definition and then specify the new job name in your schedule.

env_aliases in Plugin config #

As part of our effort to streamline the configuration experience in Meltano, we are deprecating the env_aliases attribute of plugin definitions. Previously env_aliases provided two functions:

1. Sourcing setting values from the terminal by a name other than the default environment variable (of the form <PLUGIN_NAME>_<SETTING_NAME>).
2. Writing setting values into the plugins’ runtime environment under an environment variable name other than the default.

For sourcing setting values we encourage users going forward to make use of the default environment variables (of the form <PLUGIN_NAME>_<SETTING_NAME>) for settings in most cases. These can be conveniently found for a given plugin by running meltano config <plugin> list. In cases where an environment variable of a name other than the default must be used to source a setting value, Meltano supports referencing in meltano.yml. For example:

plugins:
  extractors:
    - name: tap-gitlab
      config:
        ultimate_license: $GITLAB_API_ULTIMATE_LICENSE

This will take the value from the environment variable GITLAB_API_ULTIMATE_LICENSE and use it configure the tap-gitlab setting named ultimate_license.

For writing setting values into the plugins’ runtime environment under an environment variable name other than the default, we support the env: key for setting definitions. These can be added or overridden in your meltano.yml file. For example:

plugins:
  extractors:
    - name: tap-gitlab
      settings:
        - name: ultimate_license
          env: GITLAB_API_ULTIMATE_LICENSE

This will create an environment variable called GITLAB_API_ULTIMATE_LICENSE in the plugins’ runtime environment with the configured value of the setting ultimate_license.

Updating your Project #

Before v2.0 Meltano made use of env_aliases internally and in several common plugins. To ensure Meltano and those plugins continue to work as expected, references to the deprecated environment variables in your own project should be replaced. The table below contains details of the env_aliases that have been removed, and the corresponding default setting environment variable to use in each case. If for any reason you wish to keep sourcing or writing setting values to deprecated aliases, we recommend using the techniques detailed above to replace the functionality formally provided by env_aliases.

CLI and API Changes #

Use --state-id instead of --job_id #

In 2.0, many references to “Job ID” in our code and docs were changed to the more accurate name of “State ID”.

If you are explicitly providing a --job_id option to any scripted or otherwise automated CLI workflows, these commands should be updated to use the --state-id option instead.

Schedule list format changes #

If you have custom orchestrator integrations based on the meltano schedule list command, you will need to make adjustments to handle a new output format.

With the addition of support for scheduled jobs in 2.0, the schema output of meltano schedule list --format=json has changed. It now includes a top level field schedules and two nested array fields job and elt which hold and describe their respective schedules.

ex:

{
  "schedules": {
    "job": [
      {
        "name": "daily-doit",
        "interval": "@daily",
        "cron_interval": "0 0 * * *",
        "env": {},
        "job": {
          "name": "simple-demo",
          "tasks": [
            "tap-gitlab hide-gitlab-secrets target-jsonl",
            "tap-gitlab target-csv"
          ]
        }
      }
    ],
    "elt": [
      {legacy elt schedule entry remains unchanged}, ...
    ]
  }
}