support more extensible command-line argument parsing about kernels HOT 15 CLOSED

getopt and getopt_long are not thread-safe, so they will not work with Adaptive MPI (and possibly other runtimes). I believe argtable uses getopt, too. I don't know of any better command line parsing libraries, but I thought you might want to consider this before implementing it!

from kernels.

@SamTWhite Thanks for bringing this to our attention! I guess I can write my own argument parser. I would normally suggest pthread_once could be used here, but since AMPI is user-level threads, I doubt that it respects POSIX semantics in this respect.

from kernels.

If you want to follow the letter of the MPI standard, you should parse at process 0 and broadcast the parameters, because MPI doesn't guarantee that all of the processes will get the args (or private copies of the args). If you use that approach, getopt should be fine, even for a threaded MPI implementation like AMPI.

from kernels.

@jdinan We already do that. For example: https://github.com/ParRes/Kernels/blob/master/MPI1/Transpose/transpose.c#L223.

from kernels.

@jdinan Even if we can fix this for AMPI and FG-MPI, it is possible that we will need a thread-safe argument parser for other models where such serialization is harder to implement.

from kernels.

Which programming models are broken, such that you can't adequately synchronize, have one thread parse, and then share or broadcast the parameters? It's not that I don't believe you; I'm just curious.

from kernels.

I don't know for sure, but if any of the implementations do the weird trick where they turn main() into a vector of tasks, then it might be an issue. Of course, one could wait until after parsing to create said vector of tasks, but that is not always what the dynamic runtime systems people want to do.

Even if this is not a problem for us, there is no good argument against having a thread-safe input argument parser, since it seems that the standard options do not meet this requirement.

from kernels.

One argument against is that if a programming model is going to do such and thing and not provide a solution to the resulting synchronization problem, then how viable is it as a generally usable programming model? It could be that leaving the thread-unsafe argument parsing in place poses a good (and practical) challenge to such programming models.

from kernels.

from kernels.

My point is that if a programming model breaks long-standing assumptions about thread safety (as exemplified by the lack of reentrant getopt), then it should also provide the functionality needed to resolve the resulting concurrency issues. So, for me, you are fixing a problem that should not actually need to be fixed. And, if it does need to be fixed for some programming model, I want to know about it.

from kernels.

The problem here is getopt not being reentrant, not the lack of a serialization capabilities in some yet-to-be-determined parallel programming model.

from kernels.

I think I made my point already. If the programming model doesn't give you a way to deal with a sequence of operations that need to be serialized (argument parsing with getopt is only one such example), then there's a significant gap in the programming model.

from kernels.

About the comment by Jeff that we already follow the rule of having rank 0 do the argument parsing in MPI, that is actually not completely true. In the case of Synch_p2p, where the highest numbered rank carries out the verification test, we also let that rank do the parsing. Thanks for pointing out that that is not strictly conforming, Jim. We can easily make it so, and I'll do that.

from kernels.

Ah, yes I didn't realize that you were already parsing on only 1 rank, making the getopt issue, practically speaking, a non-issue for at least AMPI and the other runtimes that I know about.

from kernels.

The ETA for this exceeds the lifetime of the universe.

from kernels.