fd0 / machma Goto Github PK

The number of lines in the status area is variable so reading and keeping track of the processed/failed/workers area is difficult.

An example to see what I'm talking about is to run find / | machma -p 9 -- ls -alh {} and try to keep an eye on the "processed" count.

Maybe there's a better solution but what I was thinking: adding empty lines so that the status area stays in the same location. So there would always be number of lines equal to the maximum workers count plus one.

Here's an example of how I would like to use machma to replace parallel in my environment.

find . -name "*.png" | machma -- convert {} {}.jpg

Another way I've been experimenting with quotes and slashes to see if I get it to be interpreted correctly is by feeding it to echo.
find . -name "*.png" | machma -- echo {} {}.jpg

The expected result is that the original file would remain in place and a new one with the ".jpg" extension would be created.

What I get is a single file named "{}.jpg".

For example:
echo "a.jpg b.jpg" | machma -- cp $1 $2

It would be nice if the output could be controlled by using options.

The worker number and timestamp either contain data I don't need or are already provided by the data being output from my other process. These could be "off' by default and enabled with options or "on" by default and disabled with options.

It might also be nice to have an argument that accepts a keyword to act as a filter. Grepping is so easy that maybe it wouldn't be worth the effort but then the nice colorized output is lost.

I'm trying to use machma to speed up some operations in build scripts, but it always seems to hang without a TTY (which is the case when my build scripts normally run). I have reproduced this behavior inside of a docker container simply by including or omitting the -t flag.

This will hang:

$ docker run --rm golang:1.9 /bin/bash -c 'go get -u github.com/fd0/machma ; find /go -iname "*.go" | machma -- md5sum {}'

But this will not hang:

$ docker run -t --rm golang:1.9 /bin/bash -c 'go get -u github.com/fd0/machma ; find /go -iname "*.go" | machma -- md5sum {}'

The only difference in the commands is the -t flag.

$ cat foo.sh
#!/bin/sh
echo $1

$ for a in {1..10}; do for b in {1..100}; do echo $b; done | go run github.com/fd0/machma@latest -- ./foo.sh "{}" | wc -l; done
95
101
101
101
99
102
100
101
100
100

One would expect the same number of output lines on each run: 100.

Comparing two runs:

$ doit() { sort <(for b in {1..100}; do echo $b; done | go run github.com/fd0/machma@latest --no-id --no-timestamp -- ./foo.sh "{}") }
$ diff -U0 <(doit) <(doit)
--- /proc/self/fd/11	2021-12-31 02:37:51.409602436 +0100
+++ /proc/self/fd/12	2021-12-31 02:37:51.409602436 +0100
@@ -30,0 +31 @@
+35 35
@@ -33 +33,0 @@
-38 38
@@ -52,0 +53,2 @@
+56 56
+57 57
@@ -66,0 +69 @@
+71 71