feat: Provide optional lock to prevent concurrent pipeline execution about edk HOT 4 OPEN

I built a version of @aaronsteers 's option 2. It is available here. It's not pretty but it seems to do the trick. I think ultimately option 3 is the ideal; adding two extra commands to achieve this makes for an ugly pipeline command. :( Extending the EDK is beyond my capabilities at the moment though. ;)

from edk.

Featurewise, we could declare a new plugin command property that specifies only one copy can run at a time. That limit would need to be per environment, so prod would never be blocked by devtest, for instance. The challenge is that I don't know if the way we are logging commands today would work the same way it does for EL. In theory, though, this definitely could work.

A second approach could be to create a dummy "command" before and after the dbt execution runs. That dummy command would basically "take" a lock and subsequently "release" the lock. You'd probably want to build a max-age of the lock, so it could self-heal, and you probably would want to have an explicit command to "release" the lock in cases that you know that its process is not running.

A third option, and I think I like this best, would be to build the second solution into the dbt-ext plugin itself, and/or into the EDK, and have the ability to use prehooks and posthooks to do the same thing inline.

The challenge then would be where to store the lock artifact. That could be easy or hard depending on the deployment scenario.

from edk.

@aaronsteers thoughts on how we could help out with this? Likely would just be checking if the same plugin:command is already executing, right?

from edk.

This has been marked as stale because it is unassigned, and has not had recent activity. It will be closed after 21 days if no further activity occurs. If this should never go stale, please add the evergreen label, or request that it be added.

from edk.