Comments (4)
I built a version of @aaronsteers 's option 2. It is available here. It's not pretty but it seems to do the trick. I think ultimately option 3 is the ideal; adding two extra commands to achieve this makes for an ugly pipeline command. :( Extending the EDK is beyond my capabilities at the moment though. ;)
from edk.
Featurewise, we could declare a new plugin command property that specifies only one copy can run at a time. That limit would need to be per environment, so prod would never be blocked by devtest, for instance. The challenge is that I don't know if the way we are logging commands today would work the same way it does for EL. In theory, though, this definitely could work.
A second approach could be to create a dummy "command" before and after the dbt execution runs. That dummy command would basically "take" a lock and subsequently "release" the lock. You'd probably want to build a max-age of the lock, so it could self-heal, and you probably would want to have an explicit command to "release" the lock in cases that you know that its process is not running.
A third option, and I think I like this best, would be to build the second solution into the dbt-ext plugin itself, and/or into the EDK, and have the ability to use prehooks and posthooks to do the same thing inline.
The challenge then would be where to store the lock artifact. That could be easy or hard depending on the deployment scenario.
from edk.
@aaronsteers thoughts on how we could help out with this? Likely would just be checking if the same plugin:command is already executing, right?
from edk.
This has been marked as stale because it is unassigned, and has not had recent activity. It will be closed after 21 days if no further activity occurs. If this should never go stale, please add the evergreen
label, or request that it be added.
from edk.
Related Issues (20)
- bug: bad meltano_extension_sdk.logging import
- bug: Invoker.run doesn't use pre-configure env in underlying subprocess.run
- Improve docs and handling of static assets
- EDK logging HOT 1
- EDK docs site and basic tutorial HOT 3
- Evaluate/test `copier` based templates
- Created project has no `.gitignore` file HOT 1
- Downstream packages have conflicting dependencies HOT 6
- Paradigm to update the scaffold files created by `init` or `initialize`: command for `init --update` or similar HOT 6
- Docs: Add docs for files-only extensions
- Ctrl+C not passed to subprocesses properly
- Migrate Docs Website Google Analytics to GA4
- Update Branding
- Roadmap to support Pydantic v2 HOT 5
- Drop support for Python 3.7 HOT 1
- Issue run matatika lab in windows HOT 1
- `copier gh:meltano/edk my-new-extension` HOT 1
- Poetry install in `my-new-extension` returns `virtualenv: error: argument --prompt: expected one argument` HOT 1
- Meltano EDK utilizing code that is not implemented on Windows HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from edk.