Comments (4)
Hi @ChetanBhasin, thanks for this feature request. It seems like it would be a useful feature, though it would be quite a departure from the way Toast works today. We'd have to think through what it would look like and what the implications would be (if any). For now, I'll leave this issue open for further discussion.
from toast.
Here is an idea for how it might work:
- Tasks will run in parallel if and only if (a) they have
cache: false
and (b) they do not have a common descendant in the dependency graph (or the dependency subgraph containing only the tasks relevant for the current execution?). - For simplicity, tasks would run on the default bridge network, as they do today. This means parallel tasks can talk to each other, but they can also talk to other Docker containers that are not related to Toast (not great, but I'm on the fence about having Toast creating a user-defined bridge network).
This would likely be a large change, so if anyone is interested in taking this on, please discuss it with me first. I might not have the bandwidth to review such a change.
@ChetanBhasin For now, I think running multiple Toast jobs in parallel should "just work", as they would run on Docker's default bridge network and should be able to talk to each other (even without declaring any ports in the toastfile).
from toast.
@stepchowfun I like your suggestion. There are also some interesting considerations to keep in mind when running tasks in parallel. What if the parallel running tasks are interactive? And do we output the logs from all of them in the console? That could be quite annoying, IMO.
Anyway, I'm happy to brainstorm a solution and try implementing it.
from toast.
Those are great points. Right now Toast does not really support interactive tasks, because it does not attach stdin (except for the --shell
feature—speaking of that, we'd have to think of what that does in this hypothetical parallel world).
For output, Toast currently reuses its own stdout/stderr for the container. But we'd probably want to change that to do some kind of streaming, prefixing each chunk with some kind of line/header indicating which task it's from (I think I recall Docker Compose doing something like this, but I haven't used it in a while).
I am worried this could get quite complex unless we are careful. For example, maybe users would want to see only the output from one of their containers, and have everything else be logged to files. Or maybe users would want to customize how the network is configured. Etc...
FWIW, as a workaround for not having parallel tasks, I sometimes just run multiple processes in parallel within a single task. It's not ideal, but it works well enough for my simple use cases.
from toast.
Related Issues (20)
- Hidden Tasks HOT 10
- Print Task Schedule HOT 1
- toast 0.45.0 cannot print tasks HOT 2
- Add man page HOT 3
- Get error `Password: su: Authentication failure` when su cannot authenticate HOT 4
- Add Keywords To Project HOT 1
- Automatically disable cache when needed HOT 1
- Adding a PowerShell installation script HOT 5
- --shell causes "Unable to canonicalize the source directory. Reason: No such file or directory (os error 2)" HOT 2
- Remote execution via DOCKER_HOST incurs 50x ping latency. HOT 7
- When using remote DOCKER_HOST, cannot mount paths HOT 1
- License isn't recognized as MIT HOT 1
- Parallel task execution HOT 2
- Shared container/host download cache HOT 3
- 'toast'-bin collides with 'gsm' HOT 1
- PATH is overridden when using su HOT 3
- Adding an optional `net` or `network` corresponding to the `--network` flag in docker cli HOT 1
- Mount_paths on Windows not working properly HOT 4
- mount_paths can delete a directory in the container HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from toast.