I want to know why do u use FAKE_SYSTEM_LIBS/xxxxx.so
can you give me some hint? thanks

I tried emailing [email protected] (for general info, not a bug report), but got a delivery failure.

For example, the file below:
https://github.com/abhinavvishnu/matex/blob/master/src/deeplearning/tensorflow/mpi-tensorflow-alexnet.py

I suppose one could train it on GPU, and since MPI supports GPUDirect and RDMA, RDMA is automatically supported?

1. Add an example to the Getting started page

I got the following errors

2018-07-16 15:27:27.536541: W tensorflow/core/framework/op_kernel.cc:1192] Unknown: Exception: Message truncated, error stack:
MPI_Allreduce(855)..................: MPI_Allreduce(sbuf=MPI_IN_PLACE, rbuf=0x2049aaa00, count=256, MPI_FLOAT, MPI_SUM, MPI_COMM_WORLD) failed
MPIR_Allreduce_impl(712)............:
MPIR_Allreduce_intra(357)...........:
MPIC_Sendrecv(186)..................:
MPIDI_CH3U_Request_unpack_uebuf(599): Message truncated; 1536 bytes received but buffer size is 1024

Any comments!

I ‘confused with the MPI Allreduce Operator
the paper said that MaTEx-TensorFlow use the allreduce ops to synchronize each layer across ranks.
I think one AllReduce op is to reduce all the gradients generated one the same layer(also returns at the same time )
Only got the graph defined by user, I do not know which layer one node belongs to
so how do I know which gradients are on the same layer?
I mean how do I get the ordered list of reduction ops to do the layer-wise all-to-all reducetion.
I have not find some code do this work in this resouce

"Example of a MaTEx TensorFlow executing on four MPI ranks. Each rank will run a model replica and communicate
at each of the reduction points (i.e. the orange bars). Each model is initialized identically due to the broadcast operator at the
beginning (i.e the blue bar)."
https://www.researchgate.net/profile/Abhinav_Vishnu/publication/316184213/figure/fig2/AS:484334135189506@1492485664993/Fig-2-Example-of-a-MaTEx-TensorFlow-executing-on-four-MPI-ranks-Each-rank-will-run-a.png

1. We should make it pnetcdf linking optional for folks who want to use CSV or other file formats

I tested the code from /matex/src/deeplearning/tensorflow/examples/glibc_after_2.19/MNIST/tf_lenet3.py with command python tf_lenet3.py and I got an error:

Traceback (most recent call last):
  File "tf_lenet3.py", line 17, in <module>
    mnist = tf.DataSet("MNIST", normalize=255.0)
AttributeError: module 'tensorflow' has no attribute 'DataSet'

Then, I modified the code. I added from tensorflow.examples.tutorials.mnist import input_data and mnist = input_data.read_data_sets('MNIST_data', one_hot=True), removed mnist = tf.DataSet("MNIST", normalize=255.0) and I got the following error:

Traceback (most recent call last):
  File "tf_lenet3.py", line 70, in <module>
    for train_batch in range(int(len(mnist.training_data)/args.train_batch)):
AttributeError: 'Datasets' object has no attribute 'training_data'

What should I do to solve the problem?
@jeffdaily @vamatya @abhinavvishnu @charlesmsiegel @cabe1980 Thanks for your help!