Comments (6)
Thanks for opening an issue! I completely agree that this workflow would be better but I don't think it's possible fora package right now.
The selected_resources
variable tells the package which models & tests have been selected, but not which have changed since the last run. To do so would require a production manifest or info from version control.
I'll leave this issue open in case the situation changes or anyone has ideas to share.
from upstream-prod.
@franzoni315 I realised that I've been sitting on these changes for a while so, since it worked for you, I've merged them (#19) and will release a new version soon. I'll close this issue but please feel free to comment with any other questions or feedback you have.
from upstream-prod.
@franzoni315 I have something that might work for you - would you be willing to test it out? My approach only works on Snowflake and BigQuery atm.
The package can now use a dev relation if it was created or updated after the prod version. In your example above, it would use dev.model_1
until the next time prod.model_1
is updated. This can be controlled by changing a variable, so it's easy to disable if necessary.
If you'd like to give it a try, add this to packages.yml
, set the prefer_recent
variable and update your ref
macro:
packages:
- git: https://github.com/LewisDavies/upstream-prod.git
revision: prefer-recent
Let me know how it goes for you 🙂
from upstream-prod.
Hi, @LewisDavies ! Sorry for the delay, I did not see your update. I will see if I can try this next week!
from upstream-prod.
I tested it here, and it seems to be working properly! My test was to reference a table that was in my staging environment for a long time, and the compiled code returned the model in production environment. Then, I ran the model again in order to update the table, and the compiled code returned the model in staging environment, as I expected!
AFAIK your strategy was to look at the update date of the model on the database, right? It is not looking if the code of the model is different. Therefore, if a daily job updates the production table, then tomorrow the dbt compile process would choose the production table again, correct?
from upstream-prod.
Awesome, glad it worked! That's right, the package is checking when each table was last modified / created and picks the most recent one. So yes, in your example the production table would be chosen tomorrow after the daily job has run.
Does this approach make sense to you? I'm hopeful that even if the dev table needs to be updated daily, most projects will still see efficiency gains from not building it on every subsequent run. I'd love to hear of any suggestions you have or workflows this doesn't cover though.
from upstream-prod.
Related Issues (20)
- The modified "ref" macro breaks when using "dbt run-operation" HOT 2
- macro 'dbt_macro__ref' takes no keyword argument 'v' HOT 4
- Ephemeral models are not supported HOT 3
- 'dict object' has no attribute 'file_key_name' in macro ref (macros/ref.sql) HOT 4
- Error on upstream_prod fallback for bigquery ML model HOT 2
- CLI sending a large number of `True` and `False` messages to console when running `upstream-prod` on `dbt-bigquery` HOT 2
- Typo error in : raise_ref_not_found_error HOT 1
- Error When Using ref.sql Macro HOT 1
- --no-partial-parse flag fails with this library enabled on branch HOT 4
- Package not working with metrics HOT 2
- Env schemas don't seem to work with snapshots that use the `target_schema` config. HOT 1
- Dict object has no attribute `file_key_name` on Singular tests HOT 1
- Error with `prefer_recent` variable in ref.sql HOT 2
- `check_reqd_vars` Contradictory Error Message HOT 2
- BigQuery breaks in `get_table_update_ts` due to missing backticks
- Support custom databases
- --empty run option doesn't work when upstream_prod_enable is TRUE
- `ref()` returns `None` when prod table doesn't exist even if `upstream_prod_fallack: True` HOT 1
- redshift adapter.get_relation always returns null? HOT 9
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from upstream-prod.