data-mie / dbt-cloud-cli Goto Github PK

Implement CLI command for the https://cloud.getdbt.com/api/v2/accounts/{accountId}/jobs/{jobId}/ POST endpoint. https://docs.getdbt.com/dbt-cloud/api-v2#operation/updateJobById

https://docs.getdbt.com/dbt-cloud/api-v2#/operations/Update%20Project

Implement a CLI command for the https://cloud.getdbt.com/api/v2/accounts/{accountId}/projects/ GET endpoint. https://docs.getdbt.com/dbt-cloud/api-v2#operation/listProjects

Implement a --wait/--no-wait argument for waiting until process is finished.

Implement CLI command for the https://cloud.getdbt.com/api/v2/accounts/{accountId}/jobs/ GET endpoint. https://docs.getdbt.com/dbt-cloud/api-v2#operation/listJobsForAccount

See below - installed 0.3, worked fine, then 0.5, not fine!

0.30 - works fine:

(env) ernesto@Ernesto-Ongaro dbt_cloud_api % python3 -m pip install dbt-cloud-cli==0.3.0
Collecting dbt-cloud-cli==0.3.0
Using cached dbt_cloud_cli-0.3.0-py3-none-any.whl (15 kB)
Requirement already satisfied: requests in ./env/lib/python3.9/site-packages (from dbt-cloud-cli==0.3.0) (2.27.1)
Requirement already satisfied: click in ./env/lib/python3.9/site-packages (from dbt-cloud-cli==0.3.0) (8.0.3)
Requirement already satisfied: mergedeep in ./env/lib/python3.9/site-packages (from dbt-cloud-cli==0.3.0) (1.3.4)
Requirement already satisfied: pydantic in ./env/lib/python3.9/site-packages (from dbt-cloud-cli==0.3.0) (1.9.0)
Requirement already satisfied: typing-extensions>=3.7.4.3 in ./env/lib/python3.9/site-packages (from pydantic->dbt-cloud-cli==0.3.0) (4.0.1)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in ./env/lib/python3.9/site-packages (from requests->dbt-cloud-cli==0.3.0) (1.26.8)
Requirement already satisfied: certifi>=2017.4.17 in ./env/lib/python3.9/site-packages (from requests->dbt-cloud-cli==0.3.0) (2021.10.8)
Requirement already satisfied: idna<4,>=2.5 in ./env/lib/python3.9/site-packages (from requests->dbt-cloud-cli==0.3.0) (3.3)
Requirement already satisfied: charset-normalizer~=2.0.0 in ./env/lib/python3.9/site-packages (from requests->dbt-cloud-cli==0.3.0) (2.0.10)
Installing collected packages: dbt-cloud-cli
Attempting uninstall: dbt-cloud-cli
Found existing installation: dbt-cloud-cli 0.4.0
Uninstalling dbt-cloud-cli-0.4.0:
Successfully uninstalled dbt-cloud-cli-0.4.0
Successfully installed dbt-cloud-cli-0.3.0
(env) ernesto@Ernesto-Ongaro dbt_cloud_api % dbt-cloud
Usage: dbt-cloud [OPTIONS] COMMAND [ARGS]...

Options:
--help Show this message and exit.

Commands:
job
run

Then 0.5.0:
(env) ernesto@Ernesto-Ongaro dbt_cloud_api % python3 -m pip install dbt-cloud-cli --upgrade
Requirement already satisfied: dbt-cloud-cli in ./env/lib/python3.9/site-packages (0.3.0)
Collecting dbt-cloud-cli
Using cached dbt_cloud_cli-0.5.0-py3-none-any.whl (13 kB)
Requirement already satisfied: mergedeep in ./env/lib/python3.9/site-packages (from dbt-cloud-cli) (1.3.4)
Requirement already satisfied: requests in ./env/lib/python3.9/site-packages (from dbt-cloud-cli) (2.27.1)
Requirement already satisfied: click in ./env/lib/python3.9/site-packages (from dbt-cloud-cli) (8.0.3)
Requirement already satisfied: pydantic in ./env/lib/python3.9/site-packages (from dbt-cloud-cli) (1.9.0)
Requirement already satisfied: typing-extensions>=3.7.4.3 in ./env/lib/python3.9/site-packages (from pydantic->dbt-cloud-cli) (4.0.1)
Requirement already satisfied: idna<4,>=2.5 in ./env/lib/python3.9/site-packages (from requests->dbt-cloud-cli) (3.3)
Requirement already satisfied: certifi>=2017.4.17 in ./env/lib/python3.9/site-packages (from requests->dbt-cloud-cli) (2021.10.8)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in ./env/lib/python3.9/site-packages (from requests->dbt-cloud-cli) (1.26.8)
Requirement already satisfied: charset-normalizer~=2.0.0 in ./env/lib/python3.9/site-packages (from requests->dbt-cloud-cli) (2.0.10)
Installing collected packages: dbt-cloud-cli
Attempting uninstall: dbt-cloud-cli
Found existing installation: dbt-cloud-cli 0.3.0
Uninstalling dbt-cloud-cli-0.3.0:
Successfully uninstalled dbt-cloud-cli-0.3.0
Successfully installed dbt-cloud-cli-0.5.0
(env) ernesto@Ernesto-Ongaro dbt_cloud_api % dbt-cloud
Traceback (most recent call last):
File "/Users/ernesto/Dev/dbt_cloud_api/env/bin/dbt-cloud", line 5, in
from dbt_cloud.cli import dbt_cloud
File "/Users/ernesto/Dev/dbt_cloud_api/env/lib/python3.9/site-packages/dbt_cloud/init.py", line 1, in
from .command import DbtCloudRunStatus
ModuleNotFoundError: No module named 'dbt_cloud.command'
(env) ernesto@Ernesto-Ongaro dbt_cloud_api %

• Configuration instructions: env vars or cli args

According the api docs you can update a job to be "deleted" by changing the state of the job from 1 to 2. This might be the correct approach to deleting a job as opposed to sending a delete request.

Will this allow to export jobs to file and import them again? Getting users moving from single-tenant to multi-tenant dbt cloud (and vice versa) this would be 🤤

Add the ability to cancel all existing runs similar to job delete-all.

Should build on top existing https://github.com/data-mie/dbt-cloud-cli#dbt-cloud-run-cancel

https://docs.getdbt.com/dbt-cloud/api-v2#/operations/Create%20Connection

Implement a CLI command for the https://cloud.getdbt.com/api/v2/accounts/{accountId}/projects/{projectId} GET endpoint. https://docs.getdbt.com/dbt-cloud/api-v2#operation/getProjectById

Implement a CLI command for the https://cloud.getdbt.com/api/v2/accounts/{accountId}/runs/{runId}/cancel/ POST endpoint. https://docs.getdbt.com/dbt-cloud/api-v2#operation/cancelRunById

I am trying to create a job. But it is throwing an error with the execute steps parameter.
The following is my command -
dbt-cloud job create --project-id 241666 --environment-id 188496 --name "NewJob" --execute-steps ["dbt run"]

Following is the error that it is throwing:
Error: Invalid value for '--execute-steps': [dbt run]

I was initially trying to create a job using the dbt API by sending a post request which was also not working. The response shows that various parameter values in the request body are not valid. It would be great if someone could help me with the API problem as well. We are trying to automatically create jobs on the go based on a few parameters but manually doing that using API itself is not working.

Following the new version of Pydantic provides an breaking change in dbt-cloud-cli.

Running dbt-cloud job <arguments> provides the following traceback:
pydantic.errors.PydanticUserError: The fieldandconfigparameters are not available in Pydantic V2, please use theinfo parameter instead.

When specifying the dbt-cloud-host (e.g. emea.dbt.com) in the cli-arguments when starting a job, the job is started with the correct url.
BUT the loop for checking the run-state does not use the overridden host, but the default host cloud.getdbt.com, causing the check to fail with:

Traceback (most recent call last):
  File "/home/aretz/git/dbt_bi-vertrieb/.venv/bin/dbt-cloud", line 8, in <module>
    sys.exit(dbt_cloud())
  File "/home/aretz/git/dbt_bi-vertrieb/.venv/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/home/aretz/git/dbt_bi-vertrieb/.venv/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/home/aretz/git/dbt_bi-vertrieb/.venv/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/aretz/git/dbt_bi-vertrieb/.venv/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/aretz/git/dbt_bi-vertrieb/.venv/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/aretz/git/dbt_bi-vertrieb/.venv/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/aretz/git/dbt_bi-vertrieb/.venv/lib/python3.9/site-packages/dbt_cloud/cli.py", line 117, in run
    status = DbtCloudRunStatus(response.json()["data"]["status"])
TypeError: 'NoneType' object is not subscriptable

By drilling through the code, I found out, that the root problem was a <Response [401]>, while getting the run-state, because the API of cloud.getdbt.com is accessed where the used token is not valid.

Changing the cli.py (lines 109 - 114) from:

            run_get_command = DbtCloudRunGetCommand(
                api_token=command.api_token,
                account_id=command.account_id,
                run_id=run_id
            )

to

            run_get_command = DbtCloudRunGetCommand(
                api_token=command.api_token,
                account_id=command.account_id,
                dbt_cloud_host=command.dbt_cloud_host,
                run_id=run_id
            )

solves the problem. I'm double hefthanded and a python-novice and therefore not quite sure if it is the right place to fix this issue.

https://docs.getdbt.com/dbt-cloud/api-v2#/operations/Create%20Project

Implement a CLI command for the https://cloud.getdbt.com/api/v2/accounts/ GET endpoint. https://docs.getdbt.com/dbt-cloud/api-v2#operation/listAccounts

https://docs.getdbt.com/dbt-cloud/api-v2#/operations/Update%20Environment

https://docs.getdbt.com/dbt-cloud/api-v2#/operations/Retrieve%20Environment

First of all, thanks a lot for the work here, this is amazing.

I have been looking at dbt-cloud metadata query and was expecting to be able to run a query like:

dbt-cloud metadata query '{
  model(jobId: xxxxx, uniqueId:"xxxx") {
    name
    uniqueId
    type
    jobId
  }
}'

but it returned Error: Got unexpected extra argument ({ ........})

I was able to run the query by saving it in a file and using -f file_name though.

Is there a way to run a metadata query inline or is the only way with -f ? My use case was to get all the jobs with dbt-cloud job list and pipe this list with xargs to run a metadata query for each job.

If this is not possible today, would you be OK with a contribution to make it possible?

Github Secrets only available for maintainers. https://github.community/t/secrets-for-prs-who-are-not-collaborators/17712/10

Consider switching to CircleCI which supports CI run approvals.

It's unclear from the README and other artifacts in the repo (or from a google search) how to actually set up and use this package. In my case I'm working in MacOS Ventura.

I have used the command python3 -m pip install dbt-cloud-cli to install the package via terminal (running zsh) as suggested in [the blog post](https://docs.getdbt.com/blog/making-dbt-cloud-api-calls-using-dbt-cloud-cli. However, I am not able to invoke dbt-cloud from the CLI as the blog post seems to suggest.

Passing --settings-threads 4 to dbt-cloud job create results in a job with 1 thread (default). Nested pydantic args are not properly translated from click options and therefore the thread value does not get passed to the pydantic model. This affects all nested arguments in dbt-cloud job create.

>> dbt-cloud job create --project-id 26597 --environment-id 49819 --name "Test nested args" --settings-threads 4 --execute-steps "dbt seed"
{
  "status": {
    "code": 201,
    "is_success": true,
    "user_message": "Success!",
    "developer_message": ""
  },
  "data": {
    "execution": {
      "timeout_seconds": 0
    },
    "generate_docs": false,
    "run_generate_sources": false,
    "id": 48534,
    "account_id": REDACTED,
    "project_id": 26597,
    "environment_id": 49819,
    "name": "Test nested args",
    "dbt_version": null,
    "created_at": "2021-12-27T08:15:22.470119+00:00",
    "updated_at": "2021-12-27T08:15:22.470140+00:00",
    "execute_steps": [
      "dbt seed"
    ],
    "state": 1,
    "deferring_job_definition_id": null,
    "lifecycle_webhooks": false,
    "lifecycle_webhooks_url": null,
    "triggers": {
      "github_webhook": false,
      "git_provider_webhook": null,
      "custom_branch_only": false,
      "schedule": false
    },
    "settings": {
      "threads": 1,
      "target_name": "default"
    },
    "schedule": {
      "cron": "0 * * * *",
      "date": {
        "type": "every_day"
      },
      "time": {
        "type": "every_hour",
        "interval": 1
      }
    },
    "is_deferrable": false,
    "generate_sources": false,
    "cron_humanized": "Every hour",
    "next_run": null,
    "next_run_humanized": null
  }
}

https://docs.getdbt.com/dbt-cloud/api-v2#/operations/List%20Environments

dbt Cloud can be deployed either multi-tenant (i.e url is cloud.getdbt.com) or single-tenant, where a customer chooses their own url (like cloud.customer.getdbt.com) a bit more about it here

https://docs.getdbt.com/dbt-cloud/api-v2#/operations/List%20Connections

https://docs.getdbt.com/dbt-cloud/api-v2#/operations/Retrieve%20Connection

Implement a CLI command for the https://cloud.getdbt.com/api/v2/accounts/{accountId}/ GET endpoint. https://docs.getdbt.com/dbt-cloud/api-v2#operation/getAccountById

Implement CLI command for the https://cloud.getdbt.com/api/v2/accounts/{accountId}/runs/ GET endpoint. https://docs.getdbt.com/dbt-cloud/api-v2#operation/listRunsForAccount

I am trying to invoke a dbt job with multiple dbt commands but I am unable to do it. I am facing the below error:

["dbt deps", "dbt compile"]
Traceback (most recent call last):
File "C:\Users\ArunT\Anaconda3\lib\runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\ArunT\Anaconda3\lib\runpy.py", line 87, in run_code
exec(code, run_globals)
File "C:\Users\ArunT\Anaconda3\Scripts\dbt-cloud.exe_main.py", line 7, in
File "C:\Users\ArunT\Anaconda3\lib\site-packages\click\core.py", line 829, in call
return self.main(*args, **kwargs)
File "C:\Users\ArunT\Anaconda3\lib\site-packages\click\core.py", line 782, in main
rv = self.invoke(ctx)
File "C:\Users\ArunT\Anaconda3\lib\site-packages\click\core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "C:\Users\ArunT\Anaconda3\lib\site-packages\click\core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "C:\Users\ArunT\Anaconda3\lib\site-packages\click\core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "C:\Users\ArunT\Anaconda3\lib\site-packages\click\core.py", line 610, in invoke
return callback(*args, **kwargs)
File "C:\Users\ArunT\Anaconda3\lib\site-packages\dbt_cloud\cli.py", line 106, in run
run_id = response.json()["data"]["id"]
KeyError: 'id'

I am trying to use Bash Terminal in windows and Python Version 3.8.8.

I have checked the python code and it is expecting python list of string, so i tried all the possible ways to send the array in bash to that list but unable to get that working.

Is this supported ? If yes what is the appropriate way to send multiple dbt commands from Bash to dbt job run command?