palmetto / palm-dbt Goto Github PK
View Code? Open in Web Editor NEWdbt plugin for Palm CLI
License: Apache License 2.0
dbt plugin for Palm CLI
License: Apache License 2.0
Describe the bug
dbt 1.5 has renamed the environment variables for defer and state artifact usage. This means palm throws warnings on every dbt command that utilizes state or run_results
Your environment
palm
version: 2.5.3How to reproduce
Upgrade to dbt 1.5+ and run palm
Expected behavior
Palm doesn't rely on deprecated features and runs without verbose warnings
Actual behavior
Every palm command run produces warnings from dbt
Screenshots
None
Additional context
N/A
Context
Our test coverage on this plugin is not great. It would be great to take the time to test as much of the code as we can.
Is your feature request related to a problem? Please describe.
No, this is an improvement that will help keep the codebase healthy in the long-run.
Describe the solution you'd like
Implement tests for the following (in loose priority order):
Describe alternatives you've considered
Don't test the code, watch the project crumble over time.
Additional context
We are using pytest, there are already some tests around dbt_containerizer, look at palm-cli test_cli.py
for reference in re. testing commands.
Is there an existing feature request for this?
Context
When containerizing dbt, I want the ability to select the dbt version I am using
Is your feature request related to a problem? Please describe.
When I run containerize, it defaults to 0.21.0. I wouldn't catch this unless I knew to double-check the docker file.
Describe the solution you'd like
Prompt the user to enter (or at least confirm) the dbt version (if the package is getting it from somewhere else).
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.
Is there an existing feature request for this?
currently the stock base image created by palm init
does not contain deps. most of the commands like run
, test
, cycle
etc will execute deps first, so you always have fresh deps (though this does slow them down significantly). If we move deps to the build cycle, what are the pros? does this make sense?
run dbt deps
in the image build. Then move deps to a stand-alone palm command and a flag for the other commands.
this could leave you working with outdated deps, and when a new container is built the behavior might be unexpected. Especially if dbt packages are poorly pinned.
Context
What are you trying to do and how would you want to do it differently? Is it something you currently cannot do?
Use palm-dbt with dbt 1.0.0
Is your feature request related to a problem? Please describe.
When using palm-dbt with dbt 1.0.0, docker requires build args for username and password, not just env vars
Describe the solution you'd like
A clear and concise description of what you want to happen.
Update palm-dbt and containerize command to support dbt 1.0.0
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.
Is there an existing feature request for this?
Describe the bug
Your environment
palm
version: 2.1.0How to reproduce
palm model new --name "test_dim" --model-type dim
Expected behavior
palm model should generate director/files based on the template
Actual behavior
FileNotFoundError: [Errno 2] No such file or directory: '.../pdp-arbor/models/arbor/dim/dim_test_dim'
Additional context
Describe the bug
As a palm-dbt user, I expect the latest version of palm-dbt will install the macros I need to run the dbt projects. However, when I attempt to install them I get the error below.
Your environment
palm
version: 2.1.0palm-dbt
version: 0.0.4How to reproduce
python3 -m pip uninstall palm || true
python3 -m pip uninstall palm-dbt || true
pip install palm
pip install palm-dbt
cd jaffle_shop
palm init -p dbt
Expected behavior
palm install
installs macros
Actual behavior
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.8/bin/palm", line 8, in <module>
sys.exit(cli())
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/click/decorators.py", line 21, in new_func
return f(get_current_context(), *args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/palm/plugins/dbt/commands/cmd_install.py", line 39, in cli
shutil.copy(Path(macro_template_path, macro), Path(macros_path, macro))
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/shutil.py", line 415, in copy
copyfile(src, dst, follow_symlinks=follow_symlinks)
File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/shutil.py", line 261, in copyfile
with open(src, 'rb') as fsrc, open(dst, 'wb') as fdst:
FileNotFoundError: [Errno 2] No such file or directory: '/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/palm/plugins/dbt/macros/drop_branch_schemas.sql'
Screenshots
If applicable, add screenshots to help explain your problem.
Additional context
Add any other context about the problem here.
Describe the bug
The behavior of dbt run
on model build failure is to exit with a non-zero status code. The subprocess
call raises an error when ctx.obj.run_in_docker
receives the exit code 1 from dbt. Thus, for all of the commands that need to run in succession (dbt run && dbt run-operation drop_branch_schemas
) the failure is preventing the full set of commands from being run. In development, this is troublesome because failed builds are unintentionally being persisted in the target. We want idempotent runs, and we don't have them unless all models build perfectly the first time (unlikely...).
Your environment
palm
version: 2.0.1palm-dbt: 0.0.2
How to reproduce
Misspell a column name in a dbt model in the middle of a dag which is used by downstream models. Execute palm run
.
Expected behavior
I expect a failed dbt run to complete cleanly. If models fail to build, then the test cycle in palm cycle
should probably be skipped, but the cleanup step which runs run-operation drop_branch_schemas
should still complete unless the --persist
flag was provided.
Actual behavior
After your Database error in model....
message from dbt on a failed run
, you get a large stacktrace, and dbt run-operation drop_branch_schemas
never gets run.
Screenshots
...
Additional context
...
Describe the bug
I ran palm run
for the first time in a new project repo, so the plugin config walk-through ran. The prompt asks if you have prod artifacts stored locally, and I answered "N" and chose the default "target" folder for local artifacts. Right after this, an error threw on the dbtPluginConfigModel
:
Invalid config for plugin dbt: 1 validation error for dbtPluginConfigModel
dbt_artifacts_prod
Field required [type=missing, input_value={'dbt_artifacts_local': 'target'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.1/v/missing
Your environment
palm
version: 2.5.3How to reproduce
In a new dbt project, run any palm command or kick off the pluginconfig initialization and click "No" for the prompt about prod artifacts stored locally.
Expected behavior
The flow handles the answer no and simply doesn't throw an error if you have no prod artifacts stored locally.
Actual behavior
The flow threw an error and quit.
Screenshots
None
Additional context
N/A
The test branch name is hardcoded into the macro. Make it parameterized so that others can configure it for their specific setups.
example: {%- if target.database != "TEST" -%}
Try testing with a database that isn't named "TEST"
Pass name of test database in the macro as a param
"TEST" is hardcoded in the macro
Context
palm containerize
doesn't support dbt v1.4 or v1.5
Is your feature request related to a problem? Please describe.
Any dbt version that can run with palm should be supported by the containerizer.
Describe the solution you'd like
palm containerize
should not bork when inputs greater than 1.3 are provided in the walk-through. We should increase the max version for containerize to include the more recent dbt minor versions, 1.4 and 1.5 (and maybe the upcoming 1.6 depending on when this gets done).
Describe alternatives you've considered
N/A
Additional context
Is there an existing feature request for this?
Context
dbt now has 5 minor versions in V1 so palm should support all five and deprecate support below V1.
Is your feature request related to a problem? Please describe.
Palm rejects the two newest dbt minor versions, which it should support. Also, <= V0.19 is not supported by dbt any longer, so it doesn't need to be supported by palm either.
Describe the solution you'd like
Deprecate v0.19 and add support for dbt v1.4 and v1.5
Describe alternatives you've considered
N/A
Additional context
Is there an existing feature request for this?
Context
dbt just released v1.0! We want to support the latest version
Is your feature request related to a problem? Please describe.
Improvement - support the latest and greatest!
Describe the solution you'd like
Describe alternatives you've considered
Is there an existing feature request for this?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.