It would be nice to add support for a source installation of TBB. Currently I have to hack CMake.inc to make it compile.

The current version of the script is not aware of the structure of a source-based installation of TBB and is not able to find TBB_LIB, TBBMALLOC_LIB and its debug equivalents.

Similarly, it is not possible to create an installer using the -i flag, it brakes in this stage:

['cp', '-r', '/home/tiago/Work/tbb/lib/intel64', 'tbb/lib/']
cp: cannot stat '/home/tiago/Work/tbb/lib/intel64': No such file or directory
error code: 1

Hello all,
I'm trying to run icnc under FreeBSD 10 (with clang).
I tried to compile directly the sources without luck, so I reverted to Linux binary emulation.
On the INTEL site, I have a quite old version (0.5) in rpm format for ia32 architecture. They claims to have also the 0.6 but the link is broken.
Is there any plan to support natively FreeBSD ?

thanks,
Fausto

Scheduler stats currently count steps as created even if they get shipped to another process. Check for consumed_on should probably be done outside the step-instance and steps should get created only if needed locally.

The GCC team changed their C++ ABI, starting in GCC5. This will obviously lead to linker errors if using the older binary release and compiling an iCnC project with newer versions of G++. See habanero-rice/cnc-framework#9 for an example.

I think it would be helpful to include a new binary release that supports GCC5 and above.

Potentially dangerous unitialized state, I'm not sure it is portable to refer to yourself in the initialization list.

/home/yvdriess/repo/icnc/src/scheduler_i.cpp:83:56: warning: field 'm_root' is uninitialized when used here [-Wuninitialized]
              m_root( distributor::distributed_env() ? m_root : distributor::myPid() ),

The release notes states that one can "Write CnC graphs and use them in other CnC programs" but the graph translator is missing. make fib.h gives /opt/intel/cnc/1.0.002/bin/intel64/cnc: Command not found

Currently, serializing an array with a CnC::chunk object does not serialize the length for the user.

The length argument is passed by copy to the chunk constructor. It is used during packing and used internally when unpacking. However, the user still has to serialize his own length datum separately.

It would be more in line with the rest of the serialization interface to pass a reference or pointer to a length variable. Then, the length datum could be set for the user during unpacking. A small speed benefit is also expected, as it avoid having to pack/unpack the length datum twice.

Now:

           class Your_class : public serializable {
           private:
               double x;
               float  y;
               int    len;
               char*  arr; // array of lenth len
               ...
           public:
               void serialize( CnC::serializer & buf ) {
                   buf & x
                       & y
                       & len;
                   buf & CnC::chunk< char >( arr, len_arr ); // chunk declared below
                   ...
               }
           }

Suggested:

           class Your_class : public serializable {
           private:
               double x;
               float  y;
               int    len;
               char*  arr; // array of lenth len
               ...
           public:
               void serialize( CnC::serializer & buf ) {
                   buf & x
                         & y;
                   buf & CnC::chunk< char >( arr, len ); 
                   ...
               }
           }

Simple suggestion, but I had a nasty bug that I tracked down to the above assumption.

NB. s/len_arr/len in example code above. I copied the code verbatim from https://icnc.github.io/api/group__ser__exp.html

Running the CnC binary distributed using MPI (DIST_CNC=MPI) in conjunction with Intel Trace Analyser results in a bad termination message at the very end of the program. The CnC scheduling statistics get printed and the trace file gets written as usual.
This error does not seem to have any impact on the results of the program, besides the abnormal exit.

Error in question:

[0] [CnC 0] End step call_variants((sample_12,12))
[0] Steps created( [0] 256 )
[0] Steps scheduled( 249 ) inflight( 0, 0 )
[0] Steps requeued( 103 ) resumed( 157 )
[0] Messages sent( 128 ) Messages received( 150 ) Bcasts sent( 133 )
[0] [0] Intel(R) Trace Collector INFO: Writing tracefile traced_pipeline.single.stf in /home/yvdriess/coord/altpl
[0] [0] Intel(R) Trace Collector WARNING: message logging: 442 different messages, 0 (0.0%) sends without receive, 0 (0.0%) receives without send, 37 (8.4%) messages with reversed time stamps (average duration 1.427717e-03s)
[0] Attempting to use an MPI routine after finalizing MPI
[1] Attempting to use an MPI routine after finalizing MPI
[0] [CnC 0] runtimes/cnc_api/src/dist/mpi_comm/MpiCommunicator.cpp:60 assertion 'm_channel == 0' failed.
[1] [CnC 1] runtimes/cnc_api/src/dist/mpi_comm/MpiCommunicator.cpp:60 assertion 'm_channel == 0' failed.

=====================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   EXIT CODE: 6
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
=====================================================================================
APPLICATION TERMINATED WITH THE EXIT STRING: Aborted (signal 6)

Observed issue: clang++ compiler (v3.4) throws warnings when constructing item collection objects.

In file included from /home/yvdriess/opt/intel/cnc_1.0.001/include/cnc/cnc.h:44:
/home/yvdriess/opt/intel/cnc_1.0.001/include/cnc/internal/item_collection_base.h:404:17: warning: delete called on
      'CnC::Internal::item_collection_base<std::pair<std::basic_string<char>, int>, File, CnC::hashmap_tuner>::_callback' that is abstract but has
      non-virtual destructor [-Wdelete-non-virtual-dtor]
                delete (*i);

/home/yvdriess/opt/intel/cnc_1.0.001/include/cnc/cnc.h:280:9: note: in instantiation of member function
      'CnC::Internal::item_collection_base<std::pair<std::basic_string<char>, int>, File, CnC::hashmap_tuner>::~item_collection_base' requested
      here
        item_collection( context< Derived > & ctxt, const std::string & name = std::string() );

It doesn't introduce any bugs in this case, but it is easy to fix without any overhead (the object has a vtable already anyway).

Suggested fix:

index 9465a94..29f1dce 100644
--- a/cnc/internal/item_collection_base.h
+++ b/cnc/internal/item_collection_base.h
@@ -180,7 +180,7 @@ namespace CnC {
             item_type * create( const item_type & org ) const;
             void uncreate( item_type * item ) const;

-            typedef struct _callback { virtual void on_put( const T &, const item_type & ) = 0; } callback_type;
+            typedef struct _callback { virtual void on_put( const T &, const item_type & ) = 0;  virtual ~_callback() {}; } callback_type;
             void on_put( callback_type * cb );

             const Tuner & tuner() const { return m_tuner; }
diff --git a/cnc/internal/tag_collection_base.h b/cnc/internal/tag_collection_base.h
index aa92af3..8dbde5b 100644
--- a/cnc/internal/tag_collection_base.h
+++ b/cnc/internal/tag_collection_base.h
@@ -74,7 +74,7 @@ namespace CnC {
             const_iterator end() const { return m_tagTable.end(); }

             /// the callback type for writing re-usable graphs (CnC::graph)
-            typedef struct _callback { virtual void on_put( const Tag & ) = 0; } callback_type;
+            typedef struct _callback { virtual void on_put( const Tag & ) = 0; virtual ~_callback() {}; } callback_type;
             /// register a callback, called whe a tag is put
             /// not thread safe
             void on_put( callback_type * cb );

The scheduler-creation code for FIFO_SINGLE is identical to that for FIFO_STEAL. Since both specify the steal argument as true, they both result in FIFO_STEAL behavior.

https://github.com/icnc/icnc/blob/62aa3ca/src/context_base.cpp#L109-L122

Could you please the tag the git version, which want into the release 1.0.002?

I noticed that using the startup script start.sh for the SOCKET configuration is not passing the command line arguments.
e.g.: ./cnc_exe foo bar
will call start.sh with CNC_SOCKET_HOST_EXECUTABLE=pipeline, resulting in each client being started as:
./cnc_exe

Looking at PAL_GetProgname() in pal_util.cpp, I notice you do read in the entire "./cnc_exe foo bar”, but the file separates the executable name from its arguments with a ‘\0’. As this string (including the arguments) is passed as a null-terminated c-string to create an std::string, it does not ‘see’ the arguments.

I can quickly hack start.sh for my purposes, but perhaps it would be nice to pass the entire call, arguments and all, to CNC_SOCKET_HOST_EXECUTABLE? It should be just a matter of substituting the non-trailing ‘\0’ characters in the buffer just before line pal_util.cpp:220.

I build TBB and CNC separately and specify the path to TBB library using the -DTBBROOT variable. When I build the sample programs, I encountered this error:

# Makefile command that raise this error:
g++ -pthread  -std=c++0x -O3  -o jacobi2D jacobi2D.o jacobi2D_cnc.o -L/localdisk/local/lib/intel64 -lcnc -lrt -ltbb -ltbbmalloc

# error message:

/bin/ld: warning: libtbb.so.12, needed by /localdisk/local/lib/intel64/libcnc.so, may conflict with libtbb.so.2
/bin/ld: jacobi2D.o: undefined reference to symbol '_ZN3tbb6detail2r126wait_bounded_queue_monitorEPNS1_18concurrent_monitorEmlRNS0_2d113delegate_baseE'
/bin/ld: /localdisk/local/lib64/libtbb.so.12: error adding symbols: DSO missing from command line

Any ideas?

It is a little bit unusual to have to install CnC first to test it, especially for binary package builds. So it would be useful if the test could be run with the in-source libraries, too.

The tbbcompat.h header does not report the required versions string, as the #error preprocessor directive does not expand the text that follows.

#error Need TBB version #CNC_REQUIRED_TBB_VERSION_STRING or newer

produces the error:

/home/yvdriess/repo/icnc/cnc/internal/tbbcompat.h:57:2: error: #error Need TBB version #CNC_REQUIRED_TBB_VERSION_STRING or newer

For tests, I only added fib and cholesky in tests/CMakelists.txt (Comment out other tests).

After running "cmake .." in tests/build, the output seems correct:
"Warning: no $VT_ADD_LIBS env var set.
-- Configuring done
-- Generating done
-- Build files have been written to: /icnc/tests/build"
In folder tests/build/fib, I got the following files/dictionaries: "CMakeFiles CTestTestfile.cmake Makefile cmake_install.cmake"

When running "cmake test", the massage is like this
"CMake Error: The source directory "/icnc/tests/build/test" does not exist.
Specify --help for usage, or press the help button on the CMake GUI."

When running "cmake fib", the massage is like this:
"CMake Error: The source directory "/icnc/tests/build/fib" does not appear to contain CMakeLists.txt.
Specify --help for usage, or press the help button on the CMake GUI."

When enter /icnc/tests/build/fib and "make", the massage is like this:
"[ 50%] Building CXX object fib/CMakeFiles/distfib.dir/fib.cpp.o
In file included from /icnc/tests/fib/fib.cpp:38:0:
/usr/include/cnc/dist_cnc.h: In instantiation of 'CnC::dist_cnc_init<C1, C2, C3, C4, C5>::dist_cnc_init() [with C1 = fib_context; C2 = CnC::Internal::void_context; C3 = CnC::Internal::void_context; C4 = CnC::Internal::void_context; C5 = CnC::Internal::void_context]':
/icnc/tests/fib/fib.cpp:230:39: required from here
/usr/include/cnc/dist_cnc.h:589:63: error: no matching function for call to 'CnC::Internal::dist_init::dist_init(CnC::dist_cnc_init<fib_context>::mysubscriber)'
dist_cnc_init() : Internal::dist_init( mysubscriber() ) {}
^
/usr/include/cnc/dist_cnc.h:589:63: note: candidates are:
In file included from /usr/include/cnc/dist_cnc.h:502:0,
from /icnc/tests/fib/fib.cpp:38:
/usr/include/cnc/internal/dist/dist_init.h:127:13: note: CnC::Internal::dist_init::dist_init(CnC::Internal::dist_init::subscriber_type, long int, bool)
dist_init( subscriber_type subscriber, long flag = 0, bool dist_env = false )
^
/usr/include/cnc/internal/dist/dist_init.h:127:13: note: no known conversion for argument 1 from 'CnC::dist_cnc_init<fib_context>::mysubscriber' to 'CnC::Internal::dist_init::subscriber_type {aka void (*)()}'
/usr/include/cnc/internal/dist/dist_init.h:123:16: note: constexpr CnC::Internal::dist_init::dist_init(const CnC::Internal::dist_init&)
struct dist_init
^
/usr/include/cnc/internal/dist/dist_init.h:123:16: note: no known conversion for argument 1 from 'CnC::dist_cnc_init<fib_context>::mysubscriber' to 'const CnC::Internal::dist_init&'
make[2]: *** [fib/CMakeFiles/distfib.dir/fib.cpp.o] Error 1
make[1]: *** [fib/CMakeFiles/distfib.dir/all] Error 2
make: *** [all] Error 2"

Am I missing something to running the tests successfully? Thanks!

I was playing with the step priority tuning support in icnc, and I thought it was strange that I always got basically the same performance, even when I set a priority which should have really hurt the performance in my application.

I put together a very minimal test to see if priorities were doing anything. I figured if I created 10 steps and ran them on a single thread, then they would execute in order of their priority. However, my observation is that they just execute in FIFO order:

[CnC] Bind thread 0 to processor 0
[CnC] Using FIFO_SINGLE scheduler [PRIORITY ON] [PINNING ON]
Step priority is 0
Step priority is 3
Step priority is 6
Step priority is 9
Step priority is 2
Step priority is 5
Step priority is 8
Step priority is 1
Step priority is 4
Step priority is 7
Executing step 0
Executing step 3
Executing step 6
Executing step 9
Executing step 2
Executing step 5
Executing step 8
Executing step 1
Executing step 4
Executing step 7
Steps created( 10 )
Steps scheduled( 10 ) inflight( 0, 0 )
Steps requeued( 0 ) resumed( 0 )

This simple test app is added in PR #32. See the readme included with the example for instructions on how to build and run the app.

I'm using the latest binary distribution (with TBB bundled) on a FC19 x86_64 box.

When running a CnC app with MPI and ITAC enabled, it successfully executes and writes a tracfile, but when shutting down it reports

[0] [0] Intel(R) Trace Collector INFO: Writing tracefile traced_pipeline.single.stf in ...
[0] [0] Intel(R) Trace Collector WARNING: message logging: 442 different messages, 0 (0.0%) sends without receive, 0 (0.0%) receives without send, 37 (8.4%) messages with reversed time stamps (average du
ration 1.427717e-03s)
[0] Attempting to use an MPI routine after finalizing MPI
[1] Attempting to use an MPI routine after finalizing MPI
[0] [CnC 0] runtimes/cnc_api/src/dist/mpi_comm/MpiCommunicator.cpp:60 assertion 'm_channel == 0' failed.
[1] [CnC 1] runtimes/cnc_api/src/dist/mpi_comm/MpiCommunicator.cpp:60 assertion 'm_channel == 0' failed.

APPLICATION TERMINATED WITH THE EXIT STRING: Aborted (signal 6)

The current thread pinning implementation prevents CnC from building on Darwin.

[ 36%] Building CXX object CMakeFiles/cnc.dir/src/simplest_scheduler.cpp.o
In file included from /Users/yvdriess/dev/icnc/src/simplest_scheduler.cpp:36:
/Users/yvdriess/dev/icnc/src/pinning.h:139:38: error: unknown type name 'cpu_set_t'
        inline static int cpu_count( cpu_set_t * s )
                                     ^
/Users/yvdriess/dev/icnc/src/pinning.h:154:13: error: unknown type name 'cpu_set_t'
            cpu_set_t _procmask; CPU_ZERO( &_procmask );
            ^
/Users/yvdriess/dev/icnc/src/pinning.h:158:17: error: unknown type name 'cpu_set_t'
                cpu_set_t _dwProcessAffinity; CPU_ZERO( &_dwProcessAffinity );
                ^

Although Darwin prevents actual thread pinning (security, BSD is paranoid like that), it is perhaps possible to work with the new affinity API to produce a similar effect:
https://developer.apple.com/library/mac/releasenotes/Performance/RN-AffinityAPI/