Comments (3)
From dbt slack
you may make use of dbt macros for adding dynamic functionality to the config
element of your dbt model:
config(partitions = [parse_partition()], ...)
or simply do this:
https://docs.getdbt.com/docs/building-a-dbt-project/dont-nest-your-curlies
{{
config(
materialized = 'xxxxx',
properties={
"transactional": var('DBT_TRANSACTION')
}
...
)
}}
from dbt-trino.
@bachng2017 thank you for the feedback.
It is really nice to see that the new features come to use in the community.
Regarding the 1st point, please do describe in detail your use case with concrete code snippets.
This would make it easier for us to understand what you are trying to achieve.
Same thing for the 2st point.
It seems to me that the code snippet is taken out of a dbt model. What I don't quite understand is the limitation that you are speaking about.
I assume that you want properties to be evaluated dynamically. I'm not sure whether this is possible in dbt
, but i'll ask on dbt slack and get back with feedback.
from dbt-trino.
Let me clarify
- in the case we need not only transaction=true for all 10 models, but 3 models need to have partition info. What we would like to do is: set the transaction=true for
all
model in dbt_project.yml (a) and add parition to 1 models (b), which in total is2
configurations.
project_path:
+ properties:
transactional: true
model1:
+ properties:
partitioned_by = ARRAY['dt']
But with the current implementation, we need to add configuration for all 10 models separately
. In this case we could configure the global
property as above and local
property in model 1 like this, which overwrite the global
{{
config(
properties={
"transactional": "true",
"partitioned_by" = "ARRAY['dt']"
}
...
)
}}
But let image we have more models, more common
properties, the configuration will be more redundant.
Notes that in this case partition
and transaction
are only samples. The point is how efficient to write the configuration for a large number of models with common properties
and some few special cases
.
- Yes, we could use
var
ad you've describe to achieve dynamic configuretion in dbt. But defining a transactional = true/false is not enough. The code will not run in the environment where transaction is not understood at all like this one. We need a way to "remove" the property dynamically.
> create table test (id int) with (transactional='false');
Query 20211206_115505_02613_jzqcx failed: Catalog 'hive' does not support table property 'transactional'
for .2 we are now have a small patch for adapters.sql as following. It works in our case but not sure for other cases
...
227 {% macro properties(properties) %}
228 {% set _properties = {} %}
229 {% for key, value in properties.items() %}
230 {% if value is defined and value != '' %}
231 {% do _properties.update({key:value}) %}
232 {% endif %}
233 {% endfor %}
234
235 {%- if _properties | length -%}
236 WITH (
237 {%- for key, value in _properties.items() -%}
238 {{ key }} = {{ value }}
239 {%- if not loop.last -%}{{ ',\n ' }}{%- endif -%}
240 {%- endfor -%}
241 )
242 {%- endif -%}
243 {%- endmacro -%}
from dbt-trino.
Related Issues (20)
- Test grants with Starburst Enterprise
- `dbt/adapters/trino/connections.py` calls `trino.auth.CertificateAuthentication` with incorrect named parameters when upgrading to version 1.2.2
- upgrade to support dbt-core v1.3.0
- upgrade to support dbt-core v1.3.0 HOT 1
- Add support for `merge_exclude_columns` in configuring `merge` incremental models
- Analyze whether the migration of dbt_utils `current_timestamp` macros in dbt 1.3.0 affects dbt-trino HOT 1
- impersonate user config for ldap auth HOT 1
- persist_docs support for Glue HOT 7
- Enable column-level comments in views tests HOT 1
- Merge Strategy + on_schema_change seem to be fixed HOT 10
- dbt-trino should use trino=="0.319.0" (if not >=). HOT 1
- Replacing view with table fails HOT 13
- Temp/backup relations do not use model alias for table materialization HOT 4
- Insert query ID in adapter response HOT 8
- Create partitioned tables and table indexes with postgresql connector HOT 1
- Incremental materialization doesn't drop temp table HOT 2
- Support for Python models in dbt-trino HOT 2
- dbt_utils deduplicate macro not working with trino HOT 5
- Upgrade `trino-python-client` to 0.321.0 HOT 1
- Add query_id, query executed and rows_effected to adapter responses HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dbt-trino.