famzah / popen-noshell Goto Github PK

What steps will reproduce the problem?
1. create a folder/directory named /flash using popen_noshell_compat().
fp = popen_noshell_compat ("mkdir /flash", "r", &pCloseArg, 1);

2. Open a new file and copy into the folder created above using 
popen_noshell_compat().
fp = popen_noshell_compat ("cp test.txt /flash/", "r", &pCloseArg, 1);

3. Delete all contents of /flash folder using popen_noshell_compat().
fp = popen_noshell_compat ("rm -f /flash/*", "r", &pCloseArg, 1);

Although the return value of the step 3 is successful. None of the contents of 
the folder /flash is deleted. The intended result is to delete all the contents 
of /flash directory

What is the expected output? What do you see instead?
In above example all the contents of the /flash folder should have been deleted 
on this command. Instead none of the contents of /flash directory is deleted.


What version of the product are you using? On what operating system?
OS: Linux 2.6.35.6-45.fc14.i686
GCC: gcc version 4.5.1 20100924 (Red Hat 4.5.1-4) (GCC)
popen-noshell version: 1.0


Please provide any additional information below.
In above example "pCloseArg" is structure instance of "struct 
popen_noshell_pass_to_pclose".

Test program can be provided on request.

Original issue reported on code.google.com by [email protected] on 7 Jul 2014 at 11:46

Hi!

I don't (yet) have a lot of information about how it happens, but putting popen-noshell in a threaded application (some usual threads, as well as per-connection threads originating from libmicrohttpd), I occassionally see:

Fatal error in GNU libmicrohttpd daemon.c:2039: Failed to join a thread

followed by the libmicrohttpd exiting the application... Not yet sure if this is a libmicrohttpd bug, or an interference with pthreads itself. The above error message is printed with pthread_join() fails. (ESRCH is returned, and if I decoded the pthread_t correctly, it's ->tid field is zero.)

So this is just a heads-up, or maybe you have a hint right away. I'll add anything I find out of course.

Thanks,
Jan-Benedict

I'm trying to write something for an OpenWRT based router. Calling popen_noshell is definitely faster than popen, though at an expected cost. I discovered that each process opened this way will use some 8MB of RAM from the parent process. It's not leaking, but it uses that much while the process is open. What would be the reason, and is there any way to lower that?

Not an "issue" in the classical sense, but I was searching for an 
implementation of popen that does not use fork (since pthread_atfork does nasty 
things here with some libraries), and that is bidirectional (I want to 
"control" some child process).

It looks like it shouldn't be too hard to add a third option to r and w that 
allows for proper bidirectional communication.

Original issue reported on code.google.com by [email protected] on 7 Mar 2013 at 9:25

A fellow developer suggested that we could greatly simplify the critical executed code in the child process by using an intermediate "wrapper" program which would set up pipe redirection, etc. and finally call the real target binary that we wanted to exec().

Hi,

I just found this library, it is very useful for me (my main problem with the standard popen() implementation was that if I disable memory overcommit, it fails for a program with large memory usage). However, I found the following issue: if I open multiple pipes for writing (the use-case is writing compressed output with '/usr/bin/zip output.zip -'), the plose_noshell() function can hang (i.e. not return, top shows the program consuming 0% CPU, this occurs in the waipit() call at line 540). In relation, the child processes do not terminate (do not reach the 'zombie' state). Note that I use the zip program, which reads from its stdin until it receives EOF.

My guess for the cause is that the processes started later also inherit the write end of the pipes of previously started processes, resulting in that the fclose() call at line 536 does not really close the pipe as other handles also reference it in the processes which were started later. In accordance with this, adding the FD_CLOEXEC flag to the write end of the pipes created in popen_noshell() after starting the child process fixes this problem for me. This can be achieved with the following patch (applied to popen_noshell.c):

351,352d350
< int flags = fcntl(pipefd[0],F_GETFD);
< fcntl(pipefd[0],F_SETFD,flags | FD_CLOEXEC);
356,357d353
< int flags = fcntl(pipefd[1],F_GETFD);
< fcntl(pipefd[1],F_SETFD,flags | FD_CLOEXEC);

On the other hand, I'm not an expert on dealing with file descriptors (that's why I try to avoid using fork / vfork in the first place), so it can be something more complicated. Also, there might still be a problem if an other thread calls fork() or clone() between creating the pipe and setting this flag.

I will try to attach a minimal test-case if I can.

Best regards,
Daniel

What steps will reproduce the problem?
1. Call to popen_noshell_compat with an invalid command, i.e, command  which 
popen_noshell_split_command_to_argv fails.
2.
3.

Below is the fix:
Change

#define _popen_noshell_split_return_NULL { if (argv != NULL) free(argv); if 
(command != NULL) free(command); return NULL; }

to 

#define _popen_noshell_split_return_NULL { if (argv != NULL) free(argv); if 
(command != NULL) free(command); *free_this_buf = NULL; return NULL; }

Original issue reported on code.google.com by [email protected] on 21 Feb 2013 at 12:09

The original user comment can be found here:
http://blog.famzah.net/2009/11/20/a-much-faster-popen-and-system-implementation-
for-linux/#comment-2078

The referred page (http://www.unixguide.net/unix/programming/1.1.3.shtml) says 
that _exit() must be used, instead of exit(), in situations like ours. All this 
sounds reasonable, but there are a few more things to consider:
  * we use exit() only for error termination -- testing will require that we manage to fail in the respective code blocks
  * we also use err() to exit on errors -- it must be tested whether it uses exit() or _exit(), and be replaced with a custom macro if exit() is used
  * this needs testing -- I've never had a test scenario where this has caused a problem

Resolution for now is 'do-nothing' until a practical test is shown, where 
exit() gives problems.

Original issue reported on code.google.com by [email protected] on 12 Mar 2011 at 11:38

I've recently found that on my system (Ubuntu 16.04, Linux 4.4.0, gcc 5.3.1, gdb 7.11.1), if I try to debug a program using popen_noshell() calls, gdb misses breakpoints after the subprocess creation.

I've found that this can be worked around by adding either CLONE_UNTRACED or CLONE_VFORK as an extra flag to the clone() system call (as far as I understand, the first option explicitly forces gdb not to follow the newly created process, while I have no idea how the second one works).

I'm not sure if it's a bug in gdb or expected but undocumented behavior; I don't remember exactly, but it's very probable that previous gdb versions did not have this problem, but I don't have easy access to any right now to test this.

I've submitted a bug to the gdb developers which includes a minimal test case as well here https://sourceware.org/bugzilla/show_bug.cgi?id=20322 -- given the large number of bugs they have there, I decided to post here as well for better reference in case someone runs into the same problem

For background, Solaris/OpenSolaris(RIP)/Illumos have a forkx() and vforkx() that takes a flags option. One of the flags is to not send SIGCHLD to the parent when the child dies. The popen() and system() implementations in those systems do request that SIGCHLD not be sent: because if the parent happens to have a handler for that signal, it can reap the child opened by popen() (or system()), which then makes it impossible to retrieve its exit status in pclose() or system().

I've replaced a popen call I was using with this, works great except in one respect, in my original call I was using this " 2>&1" to pick up both stdout and stderr of my popened command and combine them, now it looks as though this is being ignored and stderr is being attached to the stderr of my calling process. I've looked at ignore_stderr and it is working as this describes, so this is not a bug as such.

Is it possible to detach stderr from the parent process and attach it to the stdout of my called command so I can read these back together?

This is a problem I've come across when popen-noshell is built as a shared 
library (this only happens randomly though):

Program terminated with signal 4, Illegal instruction.
#0  0x00007f7427cc3ce9 in _dl_x86_64_restore_sse () from 
/lib64/ld-linux-x86-64.so.2
(gdb) bt
#0  0x00007f7427cc3ce9 in _dl_x86_64_restore_sse () from 
/lib64/ld-linux-x86-64.so.2
#1  0x00007f7427cbd025 in _dl_fixup () from /lib64/ld-linux-x86-64.so.2
#2  0x00007f7427cc3675 in _dl_runtime_resolve () from 
/lib64/ld-linux-x86-64.so.2
#3  0x00007f7426baabc1 in popen_noshell_child_process_by_clone (raw_arg=<value 
optimized out>) at popen_noshell.c:176
#4  0x00007f7425965ccd in clone () from /lib64/libc.so.6
(gdb) up
#1  0x00007f7427cbd025 in _dl_fixup () from /lib64/ld-linux-x86-64.so.2
(gdb) up
#2  0x00007f7427cc3675 in _dl_runtime_resolve () from 
/lib64/ld-linux-x86-64.so.2
(gdb) up
#3  0x00007f7426baabc1 in popen_noshell_child_process_by_clone (raw_arg=<value 
optimized out>) at popen_noshell.c:176
176     _popen_noshell_child_process(arg, arg->pipefd_0, arg->pipefd_1, 
arg->read_pipe, arg->ignore_stderr, arg->file, arg->argv);

The problem here is that _popen_noshell_child_process is an externally visible 
symbol and thanks to "-z lazy" (which is the default for GNU ld the dynamic 
linker) tries to lazily resolve that symbol at runtime. Ordinarily this would 
be required to make things like LD_PRELOAD work.

Note that this can also happen for any of the other symbols the child process 
accesses, e.g. close(), dup2() and whatever other symbols these functions use 
internally, however most likely the libc has already resolved those symbols 
before we even get to our popen_noshell() call.

Explicitly using the "-z now" flag for ld seems to have fixed this problem for 
me.

Original issue reported on code.google.com by [email protected] on 22 Jun 2012 at 1:47

Attempting to build this fails (at least on recent Linux distro):

popen_noshell.c: In function ‘popen_noshell_vmfork’:
popen_noshell.c:254:34: error: ‘CLONE_VM’ undeclared (first use in this 
function)
popen_noshell.c:254:34: note: each undeclared identifier is reported only once 
for each function it appears in

This come from the fact than _GNU_SOURCE is set late in popen_noshell.c, and 
sched.h relies on stuff (__USE_GNU) defined in features.h for its checks, not 
on _GNU_SOURCE directly. Since features.h has most likely already been included 
by one of the other system headers, the include guards prevent this late 
_GNU_SOURCE from being useful.

Moving it to the top (before the includes) fixes it.

Original issue reported on code.google.com by [email protected] on 3 Oct 2012 at 12:53