bindscha / x-stream Goto Github PK

Hello,

I'm trying to build x-stream. I followed your instruction at INSTALL.md.
It looks like I'm missing a library. I've installed Hadoop (http://hadoop.apache.org/) and set HADOOP_HOME.
Do you have any advise how I can fix my problem?

Thanks in advance

$make
g++ -O3 -g -Wall -Wno-unused-function -Wfatal-errors -DCOMPACT_GRAPH -DZLIB_COMPRESSION_LEVEL=Z_BEST_SPEED -msse4.2 -include core/types.h -I/usr/local/include/boost-numeric-bindings -IADOOP_HOME/include -c -o object_files/core.o core/core.cpp
In file included from core/../utils/memory_utils.h:34:
from` core/autotuner.hpp:22,
from core/split_stream.hpp:21,
from core/disk_io.hpp:21,
from core/x-lib.hpp:21,
from core/core.cpp:19:
core/../utils/../core/hdfs_io.h:12:23: fatal error: hdfs/hdfs.h: No such file or directory
#include <hdfs/hdfs.h>
^
compilation terminated.
make: *** [object_files/core.o] Error 1

Hello,

I'm trying to build x-stream. I followed your instruction at INSTALL.md.
It looks like I'm missing a library. I've installed Hadoop and set HADOOP_HOME.
Do you have any advise how I can fix my problem?
Thanks in advance

$make
g++ -O3 -g -Wall -Wno-unused-function -Wfatal-errors -DCOMPACT_GRAPH -DZLIB_COMPRESSION_LEVEL=Z_BEST_SPEED -msse4.2 -include core/types.h -I/usr/local/include/boost-numeric-bindings -IADOOP_HOME/include -c -o object_files/core.o core/core.cpp
In file included from core/../utils/memory_utils.h:34:
from` core/autotuner.hpp:22,
from core/split_stream.hpp:21,
from core/disk_io.hpp:21,
from core/x-lib.hpp:21,
from core/core.cpp:19:
core/../utils/../core/hdfs_io.h:12:23: fatal error: hdfs/hdfs.h: No such file or directory
#include <hdfs/hdfs.h>
^
compilation terminated.
make: *** [object_files/core.o] Error 1

if(conductance_per_processor_data::edges_from_red >
	   conductance_per_processor_data::edges_from_black) {
	  base_edges = conductance_per_processor_data::edges_from_red;
	}
	else {
	  base_edges = conductance_per_processor_data::edges_from_black;
	}
		  
	weight_t conductance_value = 
	  ((weight_t)conductance_per_processor_data::crossover_edges)/
	  base_edges;
	BOOST_LOG_TRIVIAL(info) << "ALGORITHM::CONDUCTANCE::VALUE "
				<< conductance_value;

The base edge taken in the code is the set with the most edges in it, however if I am not mistaken it should be the set with least edges in it.

I follow the step which is described in the READ.md. But when I run the program with different datasets, there is an error--"Unable to open stream file:No such file or directory". The files named "test" and "test.ini" both existed under the "generator" folder.
The detail is:

[root@node01 bin]# ./benchmark_driver -g ../generators/test -b pagerank --pagerank::niters 10 -a -p 16 --physical_memory 268435456
<INFO 7f0e7a877700> STARTUP
<INFO 7f0e7a877700> CORE::GRAPH ../generators/test
<INFO 7f0e7a877700> CORE::ALGORITHM pagerank
<INFO 7f0e7a877700> SG-DRIVER-ORIGINAL
<WARNING 7f0e7a877700> Disk count not specified, assuming it to be 1
<INFO 7f0e7a877700> CORE::CONFIG::PROCESSORS 16
<INFO 7f0e7a877700> CORE::CONFIG::PHYSICAL_MEMORY 268435456
<INFO 7f0e7a877700> CORE::CONFIG::VERTICES 1048576
<INFO 7f0e7a877700> CORE::CONFIG::VERTEX_SIZE 12
<INFO 7f0e7a877700> CORE::CONFIG::VERTEX_BUFFER 12582912
<INFO 7f0e7a877700> CORE::CONFIG::EDGES 16777216
<INFO 7f0e7a877700> CORE::CONFIG::PARTITIONS 16
<INFO 7f0e7a877700> CORE::CONFIG::FANOUT 16
<INFO 7f0e7a877700> CORE::CONFIG::SUPER_PARTITIONS 1
<INFO 7f0e7a877700> CORE::CONFIG::STREAM_UNIT 16777216
<INFO 7f0e7a877700> CORE::CONFIG::BUFFER_SIZE 47810235
<INFO 7f0e7a877700> CORE::CONFIG::IO_NUMIOQS 1
<INFO 7f0e7a877700> CORE::CONFIG::MAX_BUFFERS 4
<INFO 7f0e7a877700> CORE::CONFIG::MAX_STREAMS 5
<WARNING 7f0e7a877700> No mount point specified for disk0, falling back to current directory
<FATAL 7f0e7a877700> Unable to open stream file:No such file or directory

When I build the environment， I encountered such a problem。

g++ -O3 -g -Wall -Wno-unused-function -Wfatal-errors -DCOMPACT_GRAPH -DZLIB_COMPRESSION_LEVEL=Z_BEST_SPEED -msse4.2 -include core/types.h -I/usr/local/include/boost-numeric-bindings -IADOOP_HOME/include -c -o object_files/core.o core/core.cpp In file included from core/../utils/memory_utils.h:34:0, from core/autotuner.hpp:22, from core/split_stream.hpp:21, from core/disk_io.hpp:21, from core/x-lib.hpp:21, from core/core.cpp:19: core/../utils/../core/hdfs_io.h:12:23: fatal error: hdfs/hdfs.h: No such file or directory #include <hdfs/hdfs.h> ^ compilation terminated. make: *** [object_files/core.o] Error 1

Do I need to have hadoop installed to get hdfs support？

Hi,
I use x-stream to run MCST algorithm. I only use a tiny benchmark as follows:

And transfer it to binary as x-stream default input format.
The command line to run it is:

But, the program has entered a dead cycle. (Other algorithms run well)

Why is there such a situation? Thank you!!

Hello, there's a input data file type trouble when I try to use your code,this is the link of the file I used http://snap.stanford.edu/data/wiki-Vote.html . This input file is a edge list file ,I want to convert it to binary edge list used the tool you provided. Unfortunately, some details of your convert program input file has a little bit different between me. So I have to make a little change in the convert file below

for line in infile:

if line[0] == '#':
    pass
else:
    vector = line.strip().split("\t")                               
    vector = list(map(int,vector))
    vector.append(random.random()) # Edge weight
    if vector[0] > vertices:                                          //I delete minus 1 operate here
        vertices = vector[0]
    if vector[1] > vertices:
        vertices = vector[1]
    edges = edges + 1
    outfile.write(s.pack(*vector))
    if add_rev_edges:
        edges = edges + 1
        tmp = vector[0]
        vector[0] = vector[1]
        vector[1] = tmp
        outfile.write(s.pack(*vector))

But when I used the converted file to run the framework, it showed me segmentation fault like below

Do you know what might be causing this problem? And is there a standard for your required binary edge list file? When I try to input some other binary edge list file it shows me unable to read property file.

Thank you very much!

When i try to make install,there is a error.I have installed lapack and follow the install.md :

g++ -O3 -g -Wall -Wno-unused-function -Wfatal-errors -DCOMPACT_GRAPH -DZLIB_COMPRESSION_LEVEL=Z_BEST_SPEED -msse4.2 -include core/types.h -I/usr/local/include/boost-numeric-bindings -IADOOP_HOME/include -c -o object_files/core.o core/core.cpp
In file included from core/../utils/memory_utils.h:34,
from core/autotuner.hpp:22,
from core/split_stream.hpp:21,
from core/disk_io.hpp:21,
from core/x-lib.hpp:21,
from core/core.cpp:19:
core/../utils/../core/hdfs_io.h:12:23: error: hdfs/hdfs.h: No such file or directory
In file included from core/autotuner.hpp:21,
from core/split_stream.hpp:21,
from core/disk_io.hpp:21,
from core/x-lib.hpp:21,
from core/core.cpp:19:
core/../utils/boost_log_wrapper.h: In destructor ‘BOOST_LOG_TRIVIAL::~BOOST_LOG_TRIVIAL()’:
core/../utils/boost_log_wrapper.h:64: error: ‘BOOST_ASSERT_MSG’ was not declared in this scope
compilation terminated due to -Wfatal-errors.
make: *** [object_files/core.o] Error 1

Ubuntu 18.04
hadoop 3.1.1

I am getting the following error

g++ -O3 -g -Wall -Wno-unused-function -Wfatal-errors -DCOMPACT_GRAPH -DZLIB_COMPRESSION_LEVEL=Z_BEST_SPEED -msse4.2 -include core/types.h -I/usr/local/include/boost-numeric-bindings -IADOOP_HOME/include -c -o object_files/core.o core/core.cpp
In file included from core/../utils/memory_utils.h:34:0,
                 from core/autotuner.hpp:22,
                 from core/split_stream.hpp:21,
                 from core/disk_io.hpp:21,
                 from core/x-lib.hpp:21,
                 from core/core.cpp:19:
core/../utils/../core/hdfs_io.h: In member function ‘void x_lib::hdfs_io::refreshReadHandle(int)’:
core/../utils/../core/hdfs_io.h:124:7: warning: unused variable ‘closeretval’ [-Wunused-variable]
   int closeretval = hdfsCloseFile(fs, fileRDmap[fd]);
       ^~~~~~~~~~~
In file included from core/core.cpp:19:0:
core/x-lib.hpp: In constructor ‘x_lib::streamIO<A>::streamIO()’:
core/x-lib.hpp:477:72: warning: ‘new’ of type ‘x_lib::x_thread’ with extended alignment 64 [-Waligned-new=]
                 workers[i] = new x_thread(config, i, cpu_state_array[i]);
                                                                        ^
core/x-lib.hpp:477:72: note: uses ‘void* operator new(std::size_t)’, which does not have an alignment parameter
core/x-lib.hpp:477:72: note: use ‘-faligned-new’ to enable C++17 over-aligned new support
In file included from core/core.cpp:20:0:
core/sg_driver.hpp: In constructor ‘algorithm::scatter_gather<A, F>::scatter_gather()’:
core/sg_driver.hpp:173:35: warning: ‘new’ of type ‘algorithm::sg_pcpu’ with extended alignment 64 [-Waligned-new=]
       pcpu_array[i] = new sg_pcpu();
                                   ^
core/sg_driver.hpp:173:35: note: uses ‘void* operator new(std::size_t)’, which does not have an alignment parameter
core/sg_driver.hpp:173:35: note: use ‘-faligned-new’ to enable C++17 over-aligned new support
In file included from core/core.cpp:21:0:
core/sg_driver_async.hpp: In constructor ‘algorithm::async_scatter_gather<A, F>::async_scatter_gather()’:
core/sg_driver_async.hpp:153:41: warning: ‘new’ of type ‘algorithm::sg_async_pcpu’ with extended alignment 64 [-Waligned-new=]
       pcpu_array[i] = new sg_async_pcpu();
                                         ^
core/sg_driver_async.hpp:153:41: note: uses ‘void* operator new(std::size_t)’, which does not have an alignment parameter
core/sg_driver_async.hpp:153:41: note: use ‘-faligned-new’ to enable C++17 over-aligned new support
Assembler messages:
Fatal error: can't create object_files/core.o: No such file or directory
Makefile:61: recipe for target 'object_files/core.o' failed
make: *** [object_files/core.o] Error 1

when I try to make, there is a error. Could you please provide a way to insatll the library.

g++ -O3 -g -Wall -Wno-unused-function -Wfatal-errors -DCOMPACT_GRAPH -DZLIB_COMPRESSION_LEVEL=Z_BEST_SPEED -msse4.2 -include core/types.h -I/usr/include/boost-numeric-bindings -IADOOP_HOME/include -c -o object_files/core.o core/core.cpp
In file included from core/../utils/memory_utils.h:34:0,
from core/autotuner.hpp:22,
from core/split_stream.hpp:21,
from core/disk_io.hpp:21,
from core/x-lib.hpp:21,
from core/core.cpp:19:
core/../utils/../core/hdfs_io.h:12:23: fatal error: hdfs/hdfs.h: No such file or directory
#include <hdfs/hdfs.h>
^
compilation terminated.
make: *** [object_files/core.o] Error 1