dspinellis / dgsh Goto Github PK

• The executables w, fft-input, pecho, secho should reside and be built in the top level directory.
• Their name should start with dgsh
• Consider merging secho with pecho.

The following command blocks

DRAW_EXIT=1 DGSH_DOT_DRAW=graphdot/compress-compare ./unix-dgsh-tools/bash/bash --dgsh example/compress-compare.sh </dev/null

libexec is the appropriate directory for programs that are not directly executed.

Have the bash shell check whether a command is dgsh compatible when processing a dgsh pipeline.
If it isn't compatible, prepend dgsh-wrap to the command for execution.
As a result, bash will invoke dgsh-wrap which will first engage in a dgsh negotiation and then exec the passed command.

Running sgsh on a file with the following produces no output.

{{
        echo hi &
        echo there &
}} | {{
        sed s/^/a/ &
        sed s/^/b/ &
}}

Same with

sgsh -c '{{ echo hi & echo there & }} | {{ sed s/^/a/ & sed s/^/b/ & }}'

• By default list only matching elements.
• If exactly one -L -l or -v -o -c etc is specified, output only what is specified.
• If --matching-lines is specified in addition to an option in the above list, also output the corresponding matching lines.
• If more than one of the -L -l or -v -o -c etc options is specified generate one output per option.
• If more than one -e option is specified, output one stream per option (do not implement this if internally the REs are merged into one).

If neither its input channel nor its output channel is connected with a pipe, a command should not enter the negotiation procedure since there is no one to negotiate with.
The call to the dgsh negotiation library should return immediately.

• If <| arguments are given, these get assigned the available channels; stdin gets nothing
• A new option flag can ask stdin to be included to channel assignments in order to provide compatibility with example/ft2d.sh

Add comments (in the specific format used in the examples) for the new commands: fft-block8, manage-results, reorder-columns

The following command works fine.

dgsh -c 'echo hi | {{  cat & echo there & }} | cat'

The following command hangs.

dgsh -c 'echo hi | {{  cat & printf there & }} | cat'

This may help getting around build failures on Cygwin.

When running example/5x5.sh with DGSH_DOT_DRAW the following file is created.

digraph {
        n0 [label="bash --dgsh-negotiate -c matrix"];
        n0 -> n1;
        n1 [label="scatter-gather"];
}
digraph {
        n0 [label="bash --dgsh --dgsh-negotiate /tmp/dgsh-parallel-31690"];
        n0 -> n1;
        n1 [label="scatter-gather"];
}
digraph {
        n0 [label="bash --dgsh --dgsh-negotiate /tmp/dgsh-parallel-31726"];
        n0 -> n1;
        n1 [label="paste"];
}
digraph {
        n0 [label="bash --dgsh --dgsh-negotiate /tmp/dgsh-parallel-31732"];
        n0 -> n1;
        n1 [label="paste"];
}
digraph {
        n0 [label="bash --dgsh --dgsh-negotiate /tmp/dgsh-parallel-31728"];
        n0 -> n1;
        n1 [label="paste"];
}
digraph {
        n0 [label="bash --dgsh --dgsh-negotiate /tmp/dgsh-parallel-31729"];
        n0 -> n1;
        n1 [label="paste"];
}
digraph {
        n0 [label="bash --dgsh --dgsh-negotiate /tmp/dgsh-parallel-31730"];
        n0 -> n1;
        n1 [label="paste"];
}

Consider the following command

#!/usr/bin/env dgsh

row()
{
  dgsh -c "dgsh-parallel -n 5 'echo C{}' | paste"
}

matrix()
{
  dgsh -c "/home/dds/libexec/dgsh/dgsh-parallel -n 5 row"
}

export -f row matrix

call matrix | cat

1. It won't work without the explicit specification of the dgsh-parallel path.
2. With the absolute path it hangs after outputting the following.

[dds@stereo dgsh]$ example/5x5.sh
▒▒▒▒▒▒▒▒▒7▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒7/home/dds/libexec/dgsh/bash --dgsh --dgsh-negotiate /tmp/dgsh-parallel-14282

I realize that this can be fixed by attaching cat in the matrix statement, but we should somehow error and abort when dgsh commands hanve dangling I/O.

Create a shell builtin call function-name that wraps sgsh-wrap bash -c 'function-name ...'

This should get passed to the tool configuration.

Otherwise, when wrapped or modified commands are invoked, in a non-dgsh context, they attempt to negotiate and hell breaks loose. This must be solved in conjunction with #21.

Graphs that include multipipe block to mutipipe block connections, such as the following one, hang at the negotiation phase.

{{
echo hi &
echo there &
echo stranger &
}} |
{{
cat -n &
dd &
}}

This can be done by dynamically creating the processing script. Use parallel logresolve from the old version as an example.

A notes column in the index.html file should suffice. What is needed is a one-sentence explanation on the allocation of input and output channels.

The documentation for this example makes it clear that text-properties.sh
is meant to work for either one filename argument or none.
But I get an "ambiguous redirect" error if I do not give this script a filename
on the command line.

This change made it so that it works for me in both cases:

``
@@ -52,7 +52,7 @@
export -f ranked_frequency
export -f ngram

-tee <$1 |
+tee < "${1-/dev/fd/0}" |
{{
# Split input one word per line
tr -cs a-zA-Z \n |
``

Or, you might prefer using 'cat $1 | ...'

-- Guy Shaw

The merge at 86cb776 introduces a tricky bug in MacOSX (testing on other platforms is required) when executing examples/code-metrics.sh.
A dgsh-tee command suddenly exits the negotiation with error, but no other cause can be traced prior to this event.

Consider the following example, which freezes, because cat never negotiates.

$ dgsh -c 'echo hi | cat -n'
getopt: invalid option -- 'n'
Usage: cat [-u] [file ...]

Here is a sketch of the proposed solution.

• Modify dgsh_negotiate to set negotiation_completed to true
• Add the following functions.

#include <stdbool.h>
#include <stdlib.h>

bool negotiation_completed = false;
static void
on_exit_handler(int v, void *ptr)
{
  if (negotiation_completed)
    return;
  // Run negotiation indicating a failure
}
    
static __attribute__((constructor))
install_on_exit_handler(void)
{
  on_exit(dgsh_on_exit_handler);
}

The following command hangs.

/usr/local/sgsh/bin/bash --sgsh -c '{{ echo a & echo b & }} | cat  | {{ dd of=1 & dd of=2 & }} '

$ dgsh -c 'dgsh -c "{{ echo hi & echo there & }}" | paste'
/home/dds/libexec/dgsh/bash: line 1: 16110 Done                    dgsh -c "{{ echo hi & echo there & }}"
     16111 Segmentation fault      (core dumped) | paste

The second command (e.g. paste or cat) crashes.

The following command produces the output in the current directory.

DRAW_EXIT=1 DGSH_DOT_DRAW_EXIT=graphdot/reorder-columns.dot ./unix-dgsh-tools/bash/bash --dgsh example/reorder-columns.sh

• Move source code under src.
• Do not copy files into test/negotiate
• Integrate diverse tests into one framework

This will exec $INSTPREFIX/libexec/dagsh/bash --dagsh "$@".

This will invoke sgsh-tee with the right parameters.

• Use negotiate.h for implementation details
• Use dgsh.h for the public interface

On which branch do modifications take place, and what is the procedure for merging upstream changes?

I would expect a dgsh branch where upstream changes are merged from master.

I know this is not the way to do it, but we shouldn't segfault.

[dds@stereo dgsh]$ example/5x5.sh
example/5x5.sh: line 8:  2889 Done                    row
      2890 Segmentation fault      (core dumped) | cat
[dds@stereo dgsh]$ cat example/5x5.sh
#!/usr/bin/env dgsh

row()
{
  echo hi
}

row | cat

$ cat /etc/motd
Usage cat [-b size] [-i file] [-IMs] [-o file] [-m size] [-t char]
-b size Specify the size of the buffer to use (used for stress testing)
-f      Overflow buffered data into a temporary file
-I      Input-side buffering
-i file Gather input from specified file
-m size[k|M|G]  Specify the maximum buffer memory size
-M      Provide memory use statistics on termination
-o file Scatter output to specified file
-p d1[,d2...]   Permute inputs to specified outputs
-s      Scatter the input across the files, rather than copying it to all
-T dir  Specify directory for storing temporary file
-t char Process char-terminated records (newline default)

• The sgsh path needs to be set differently from PATH=/usr/local/sgsh/bin:$PATH?
• The command should exec /bin/cat in some cases?

These will invoke sgsh-tee with the right parameters.

Consider the following error report.

ERROR: No solution was found to satisfy the I/O requirements of the participating processes.dgsh-wrap: dgsh negotiation failed for /usr/bin/perl /home/dds/libexec/dgsh/dgsh-merge-sum with status code 4

dgsh-wrap: dgsh negotiation failed for /usr/bin/uniq -c with status code 4

dgsh-wrap: dgsh negotiation failed for /usr/bin/tr -s  \t\n\r\f with status code 4

dgsh-wrap: dgsh negotiation failed for /usr/bin/uniq -c with status code 4

dgsh-wrap: dgsh negotiation failed for /usr/bin/tr -s  \t\n\r\f with status code 4

dgsh-wrap: dgsh negotiation failed for /usr/bin/uniq -c with status code 4

dgsh-wrap: dgsh negotiation failed for /usr/bin/tr -s  \t\n\r\f with status code 4

dgsh-wrap: dgsh negotiation failed for /usr/bin/uniq -c with status code 4

dgsh-wrap: dgsh negotiation failed for /usr/bin/tr -s  \t\n\r\f with status code 4

dgsh-tee: dgsh negotiation failed with status code 4

The first line should offer more explanations. The remaining lines should not be outputted.

Having them in a Makefile hinders their readability and makes them brittle.

Would you like to add more error handling for return values from functions like the following?

• fputs ⇒ escape
• strdup ⇒ main

In index.html have one list for prerequisites required for compiling and installing dgsh and another with the prerequisites required for testing it.

These tests run the provided examples and see that the results are correct.

• Implement them in unix-dgsh-tools/install-wrapped.sh
• Document commands in index.html