Comments (10)
Ouch! @mfragkoulis I think I know what's going on. When running within dgsh, wrapped commands are in the path and hell breaks loose. Correct?
from dgsh.
@dspinellis This doesn't seem to be the problem this time, but we'd better check that there is no issue with wrapped commands when working inside dgsh. I'll come up with a proper example for this.
Regarding the issue at hand, dgsh
forwards a pipe that connects the standard input of a dgsh
script into the script's contents in order to form one negotiation graph.
In the example that @trantor provided, the graph of commands that dgsh should form is the following.
cat /tmp/testfile |
tee |
{{
grep -F 'info: header Subject: ' |
grep -P ' from=<\Q'"${MITTENTE}"'\E>' |
grep -oP '[[:xdigit:]]+(?=: info: header Subject:)' |
sed 's/^/^/;'
}} |
cat
Without this arrangement, programs that combine inline commands with scripts wouldn't work. There would be multiple negotiation processes nested within one another. The nested ones, such as the negotiation among the commands in match1.dgsh
, would have to wait for the outer ones to complete.
In the example at hand, the negotiation between cat /tmp/testfile | dgsh -x match1.dgsh [email protected]
would happen first. At completion, cat /tmp/testfile
would start its normal execution along with the unpacking of the match1.dgsh
script. The negotiation among the commands of the script would happen then.
Let aside the dependencies that emerge between multiple negotiation processes, which are opposite to dgsh
's rationale, this arrangement wouldn't work because cat /tmp/testfile
will transmit data that is only garbage for the negotiation process among the commands of the match1.dgsh
script and will break it.
The exact problem seems to be that dgsh
naively connects the inherited pipe to the first command in the script, which is a mere assignment. I'll look into the specifics and post a fix hopefully.
@trantor If you can work around the assignment and shift, you can probably continue your work.
from dgsh.
@mfragkoulis It's not really a problem for me, since I am not planning to run the pipeline from an interactive session of dgsh. To be honest I noticed by chance, but I thought it would be something useful to report.
I didn't quite understand your explanation about the multiple negotiation processes, I'll have to admit.
A question for both of you.
Here's a larger sample of my programme
#!/usr/local/bin/dgsh
MITTENTE=$1
shift
tee|
{{
grep -F 'info: header Subject: ' |
grep -P ' from=<\Q'"${MITTENTE}"'\E>' |
grep -oP '[[:xdigit:]]+(?=: info: header Subject:)' |
sed 's/^/^/;' &
{{
{{
/opt/zimbra/bin/zmprov -l getAllDomains |
sed 's/^/@/; s/$/#/;' &
grep -oP ': [[:xdigit:]]+: to=<[^>]+' |
sed -r 's/^: //; s/: to=</ /; s/$/#/;' |
LC_ALL=C sort &
}} |
grep -F --matching-lines --file=- |
sed "s/#$//;" &
grep -oP ': [[:xdigit:]]+: message-id=<[^>]+' |
sed -r 's/^: //; s/: message-id=</ /;' |
LC_ALL=C sort &
}} |
LC_ALL=C join -j 1
}} |
grep -E --matching-lines --file=- |
sort
It goes on, but what comes later is beside the point, on top of complicating things transforming line data into a json structure which is then transformed as well, etc. etc.
The bit with the /opt/zimbra/bin/zmprov
command is basically something which is not meant to handle standard input but just to output something, which is different from the other pipeline at the same level. Actually the command itself could receive commands on its standard input, but obviously that's not the intended objective here. It works as expected (perhaps because I provided the actual command to /opt/zimbra/bin/zmprov
as an argument so it didn't read anything from stdin, but I am uncertain about that) and I need it there to provide the pattern file input for the grep instance downstream, but I don't know how to have the command read stdin input from somewhere else (I tried something like < /dev/null
but I got an error) or just tell dgsh that it shouldn't connect the standard input of the command with the upstream flow. Not sure about my terminology but I hope I got the meaning across.
from dgsh.
To specify that a program takes no input you explicitly use dgsh-wrap -i 0
in front of it.
from dgsh.
Consider trying MITTENTE=$1 </dev/null
as a workaround. If this works, document it as a bug in the dgsh man page and close the issue.
from dgsh.
MITTENTE=$1 </dev/null
did not work, but I instructed bash to pipe input to a dgsh script only to the first pipeline within the script, not to standalone commands like MITTENTE=$1
. Respectively for output. This arrangement is satisfying for most cases.
from dgsh.
from dgsh.
tar cf - . | ( cd /tmp; tar xvf -)
has been running fine in dgsh
since we wrapped various types of commands, e.g. subshell, that dgsh
views as black box.
More specific explanation regarding input and output connections set up by dgsh
:
dgsh
needs to set up input and output connections in a pipeline of commands starting with the first command on the left for the negotiation process to work. It may in fact turn out that some commands in the graph require no input or produce no output, but during the negotiation process those commands have both their input and output channel connected (except if they are located at the edges of the graph) in order to exchange messages.
I think that the present discussion is adequate documentation.
from dgsh.
I'm worried about making dgsg more difficult to understand through special cases. If this behavior is special shouldn't we document it in the man page? Could we perhaps connect even non-pipelined commands during the negotiation phase?
from dgsh.
It was a bug not a special case.
from dgsh.
Related Issues (20)
- Avoid nout=0 on last pipe component HOT 6
- Optimize dgsh program detection
- Provide way to detect whether a script is running under dsgh? HOT 7
- Wrap rather than modify the source code of single-output commands HOT 1
- Document POSIX tools with dgsh capabilities
- Simple scripts failing with random "dgsh: timeout for negotiation" HOT 32
- Problem with grep (maybe) HOT 18
- [RFE] Parallelize processes receiving different input streams HOT 8
- Bash raising 'no version information available' HOT 4
- error: 'dir_fd' may be used uninitialized in this function HOT 6
- Executable stack HOT 2
- Unable to Build on MacOs Mojave HOT 4
- How to background a process with dgsh HOT 1
- different example HOT 5
- rsync: getaddrinfo: translationproject.org 873: Temporary failure in name resolution HOT 2
- Named pipelines HOT 1
- Porting for 1) Natively on termux (Follows aarch64) 2)Ubuntu arm64 /(any other linux distro) running inside proot container on termux HOT 21
- can not compile - gnulib problem HOT 11
- Deadlock in dgsh-tee
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dgsh.