laristra / flecsi Goto Github PK

Add build support for MPI runtime and implement mpi_execution_policy_t type.

The following header files are not in the install directory for legion:

• lowlevel_config.h
• realm/realm_config.h (basically any realm header)
• common.h
• accessor.h
• arrays.h
  etc...

also from the legion/runtime/legion directory

• legion_constraint.h

Basically it looks like any header file in the legion/runtime directory is not included.

If the goal of mesh::compute_graph_partition() is to generate the Distributed CSR format as in ParMetis, the current implementation of mesh_graph_partition::partition member is wrong. The current implementation lists the number of elements in each partition instead of the indices to the "global" sequence of the elements. For example, partitioning the Sagittarius mesh of 4 cells into two equal partitions, the mesh_graph_partition::partition should be [0, 2, 4] instead of [0, 2, 2] as currently implemented.

Create and switch to a user in the Dockerfile so we can run "make test" without error in MPI build.

Let me know if you need any guidance on the build system...

What is the purpose of ASSERT_DEATH in the zip.cc? Is it expecting the test program to crash?

On my system, if I compile it with Release or RelWithDebInfo, the test program does not crash:

/home/ollie/git/flecsi/flecsi/utils/test/zip.cc:52: Failure
Death test: zip(a, b, c_fail)
Result: failed to die.
Error msg:

Is this the correct behavior or not?

If the goal of recent changes to project.cmake and packages.cmake is to use CMAKE_PREFIX_PATH instead of TPL_INSTALL_PREFIX, the cmake rules for linking to ExodusII and LAPACKE should also be changed accordingly. Otherwise, the dependencies are not linked.

$ grep TPL_INSTALL_PREFIX *
packages.cmake:  if(EXISTS ${TPL_INSTALL_PREFIX}/include/exodusII.h
packages.cmake:     AND EXISTS ${TPL_INSTALL_PREFIX}/lib/libexodus.a)
packages.cmake:    set(IO_LIBRARIES ${TPL_INSTALL_PREFIX}/lib/libexodus.a
packages.cmake:      ${TPL_INSTALL_PREFIX}/lib/libnetcdf.a
packages.cmake:      ${TPL_INSTALL_PREFIX}/lib/libhdf5_hl.a
packages.cmake:      ${TPL_INSTALL_PREFIX}/lib/libhdf5.a
packages.cmake:      ${TPL_INSTALL_PREFIX}/lib/libszip.a
packages.cmake:      ${TPL_INSTALL_PREFIX}/lib/libz.a
packages.cmake:    include_directories( ${TPL_INSTALL_PREFIX}/include )
packages.cmake:  if(EXISTS ${TPL_INSTALL_PREFIX}/include/lapacke.h
packages.cmake:     AND EXISTS ${TPL_INSTALL_PREFIX}/lib/liblapacke.a)
packages.cmake:    include_directories(${TPL_INSTALL_PREFIX}/include)
packages.cmake:          ${TPL_INSTALL_PREFIX}/lib/liblapacke.a
packages.cmake:          ${TPL_INSTALL_PREFIX}/lib/liblapack.a
packages.cmake:          ${TPL_INSTALL_PREFIX}/lib/libblas.a
project.cmake:set( TPL_INSTALL_PREFIX ""
project.cmake:if (NOT TPL_INSTALL_PREFIX STREQUAL "")
project.cmake:  message(WARNING "TPL_INSTALL_PREFIX is obsolete, use CMAKE_PREFIX_PATH instead (and rebuild the latest version third-party libraries)")
project.cmake:  list(APPEND CMAKE_PREFIX_PATH "${TPL_INSTALL_PREFIX}")

Update build system with legion support and implement legion_execution_policy_t type.

@tuxfan : Flecsi builds fine with gcc-5, but build with gcc-4.9 is broken:
https://travis-ci.org/junghans/flecsi/jobs/124569585

Do we want to drop support for gcc-4.9?

Try to get better performance on vector accesses...

flecsi should not install gtest/gmock libraries and headers as this will conflict with all other packages using gtest/gmock.

Check the container with different compilers and change protection setting at Cmake according to the working compiler's versions.

Fresh clone fails because of what appears to be relative paths to submodules if one follows the README.md:

cmalone@varan:~/code$ git clone --recursive [email protected]:flecsi/flecsi.git flecsi-la
Cloning into 'flecsi-la'...a': git clone --recursive [email protected]:flecsi/flecsi
remote: Counting objects: 6079, done.
remote: Compressing objects: 100% (99/99), done.
remote: Total 6079 (delta 45), reused 0 (delta 0), pack-reused 5980
Receiving objects: 100% (6079/6079), 2.54 MiB | 1.60 MiB/s, done.
Resolving deltas: 100% (4292/4292), done.
Checking connectivity... done.
Submodule 'cinch' ([email protected]:flecsi/cinch.git) registered for path 'cinch'
Cloning into 'cinch'...
ERROR: Repository not found.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
fatal: clone of '[email protected]:flecsi/cinch.git' into submodule path 'cinch' failed

If instead one clones from losalamos/flecsi.git, which is the real repository, then you get things correct. Either the README.md needs updated, or the relative paths need updated.

Also note, my fork fails in the same way, effectively rendering it useless unless I go in and manually change the submodule location - or I recursively make forks of all the submodules - neither of which I should need to do...

Cloning into 'cinch'...
ERROR: Repository not found.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
fatal: clone of '[email protected]:cmsquared/cinch.git' into submodule path 'cinch' failed

The partition test gives a different result when using a different versions of scotch.
scotch-5.1 - fails:
https://travis-ci.org/junghans/flecsi/jobs/124569586
scotch-6 - passed:
https://travis-ci.org/losalamos/flecsi-third-party/builds/124566852

Should we drop support for scotch-5?

There is a mysterious warning message saying some enum class is to small:

flecsi/Build/include/flecsi/tree/tree_topology.h:1092:20: warning: ‘flecsi::tree_topology_dev::tree_branch<long unsigned int, 2ul>::action_’ is too small to hold all values of ‘enum class flecsi::tree_topology_dev::action’ action action_ : 2;

What is that about? Can it be change to some larger number?

We need meshes in an array of sizes to test scalability of FleCSI. A combination of structured and unstructured meshes of different sizes and complexities will be useful.

Then specializations could be moveable

There is a problem with corners surrounding a vertex. To reproduce, update to the latest version of
src/specializations/burton/test/burton.cc
and look at the output from the last test. The first vertex reports that it has all of the corners, and it has corner 0 twice. The output you should see is below. The corner ids for all other vertices seem reasonable.

corners_for_vertices

vertex: 0
corner: 0
corner: 0
corner: 1
corner: 2
corner: 3
corner: 4
corner: 5
corner: 6
corner: 7
corner: 8
corner: 9
corner: 10
corner: 11
corner: 12
corner: 13
corner: 14
corner: 15

vertex: 1
corner: 1
corner: 4

vertex: 2
corner: 5

vertex: 3
corner: 3
corner: 8

vertex: 4
corner: 2
corner: 7
corner: 9
corner: 12

vertex: 5
corner: 6
corner: 13

vertex: 6
corner: 11

vertex: 7
corner: 10
corner: 15

vertex: 8
corner: 14

There are several references to flexi in the current flecsi code base.

driver/flecsi-serial.cc:#include <flexi/utils/common.h>
driver/flecsi-serial.cc: return flexi::execution_t<>::execute_task(driver, argc, argv);
examples/01_simple_task/types.h:#include <flexi/specializations/burton/burton.h>
examples/01_simple_task/types.h:#include <flexi/execution/task.h>
examples/01_simple_task/types.h:using burton_mesh_t = flexi::burton_mesh_t;
examples/01_simple_task/types.h:using flexi::persistent;

Interesting enough, they are not caught by any test suites because the executables are not built.

Cloning into 'flecsi'...
remote: Counting objects: 3923, done.
remote: Compressing objects: 100% (42/42), done.
remote: Total 3923 (delta 20), reused 0 (delta 0), pack-reused 3881
Receiving objects: 100% (3923/3923), 1.26 MiB | 1.55 MiB/s, done.
Resolving deltas: 100% (2740/2740), done.
Checking connectivity... done.
Submodule 'cinch' ([email protected]:losalamos/cinch.git) registered for path 'cinch'
Cloning into 'cinch'...
remote: Counting objects: 1430, done.
remote: Compressing objects: 100% (534/534), done.
remote: Total 1430 (delta 850), reused 1428 (delta 848), pack-reused 0
Receiving objects: 100% (1430/1430), 1.22 MiB | 1.45 MiB/s, done.
Resolving deltas: 100% (850/850), done.
Checking connectivity... done.
Submodule path 'cinch': checked out '983233d2f55e8dedfab4b055158de214a534a146'
Submodule 'gtest' ([email protected]:losalamos/gtest.git) registered for path 'gtest'
Cloning into 'gtest'...
ERROR: Repository not found.
fatal: Could not read from remote repository.

Add push notifications to FleCSI project.

So how do we want to do this? Do we want to have different classes for each runtime controlled by compiler-time switches, or do we want to have each runtime specific context inherit from a parent context_t?

Develop some unit tests to cover the execution framework.

Look at the possibility of defining a new id_t with local/global/partition information available through its interface.

On Moonlight, I am getting the following error:

/lustre/scratch1/turquoise/certik/repos/flecsi/src/specializations/burton/test/burton_gradients.cc:84:29: fatal error: lapacke/lapacke.h: No such file or directory

Which can be fixed by:

--- a/src/specializations/burton/test/burton_gradients.cc
+++ b/src/specializations/burton/test/burton_gradients.cc
@@ -81,7 +81,7 @@ protected:

 // this may not be a portable way to point to lapacke.h. it works on darwin.
 #include <ccomplex>
-#include <lapacke/lapacke.h>
+#include <lapacke.h>

 const int N = 10;

But then I get:

[ 18%] Linking CXX executable test/specializations/burton_gradients
/usr/bin/ld: cannot find -llapacke
collect2: error: ld returned 1 exit status

which I don't know how to fix, because the third-party didn't install any lapacke.so:

$ ls $TPL_INSTALL_PREFIX/lib
cmake            libhdf5_hl.a      libnetcdf.settings  libscotchmetis.a
libexodus.a      libhdf5.settings  libscotch.a         libszip.a
libexoIIv2for.a  libmetis.a        libscotcherr.a      libz.a
libhdf5.a        libnetcdf.a       libscotcherrexit.a  pkgconfig

Here are my copy & paste instructions to build on Moonlight:

bash
module load friendly-testing git gcc cmake/3.4.1

export TPL_INSTALL_PREFIX=$HOME/ext/flecsi-libs

git clone https://github.com/flecsi/third-party.git flecsi-thirdparty
cd flecsi-thirdparty
mkdir build
cd build
cmake -DCMAKE_INSTALL_PREFIX=$TPL_INSTALL_PREFIX \
    -DENABLE_EXODUS=ON \
    -DENABLE_LAPACK=ON \
    -DENABLE_LAPACK=ON \
    -DENABLE_METIS=ON \
    -DENABLE_SCOTCH=ON \
    ..
make -j16


git clone --recursive -b beta-0.0-603-ge2dd9992 https://github.com/flecsi/flecsi.git
cd flecsi
mkdir build
cd build
../arch/developer-gnu
make # FAILS as indicated above
make test

cmake has exponential function now (using <<):
https://cmake.org/cmake/help/v3.0/command/math.html

@tuxfan: Should be use that instead of bc?

@ollielo : Can you install scotch-devel and metis-devel using yum on your fedora and test if flecsi builds with them?

Investigate various partitioning tools for generating primary partitions. Metis, Scotch, etc...

Segfaults when run with all other tests, but works fine when run individually

FindMetis.cmake in fbc362d does not think the libmetis.a from TPL is suitable.

-- Checking for module 'metis'
--   Package 'metis' not found
-- Could NOT find METIS: Found unsuitable version "/opt/flecsi_third_party/lib/libmetis.a", but required is at least "5.1" (found /opt/flecsi_third_party/lib/libmetis.a)

However, the header file does identify itself as version 5.1.0

[ollie@salami Build]$ grep METIS_VER /opt/flecsi_third_party/include/metis.h 
#define METIS_VER_MAJOR         5
#define METIS_VER_MINOR         1
#define METIS_VER_SUBMINOR      0

Cloning using https link fails:

git clone --recursive https://github.com/flecsi/flecsi.git
Initialized empty Git repository in /local/home/charest/flecsi/.git/
remote: Counting objects: 3722, done.
remote: Compressing objects: 100% (112/112), done.
remote: Total 3722 (delta 57), reused 0 (delta 0), pack-reused 3610
Receiving objects: 100% (3722/3722), 1.24 MiB | 1.59 MiB/s, done.
Resolving deltas: 100% (2568/2568), done.
Submodule 'cinch' ([email protected]:losalamos/cinch.git) registered for path 'cinch'
Initialized empty Git repository in /local/home/charest/flecsi/cinch/.git/
remote: Counting objects: 1426, done.
remote: Compressing objects: 100% (536/536), done.
remote: Total 1426 (delta 848), reused 1420 (delta 842), pack-reused 0
Receiving objects: 100% (1426/1426), 1.22 MiB | 1.57 MiB/s, done.
Resolving deltas: 100% (848/848), done.
Submodule path 'cinch': checked out '9798489475ba49cdd38eb91663fa0e6044ae11ce'
Submodule 'gtest' ([email protected]:losalamos/google/googletest.git) registered for path 'gtest'
Initialized empty Git repository in /local/home/charest/flecsi/cinch/gtest/.git/
fatal: remote error:
losalamos/google/googletest is not a valid repository name
Email [email protected] for help
Clone of '[email protected]:losalamos/google/googletest.git' into submodule path 'gtest' faile

Cloning using ssh link fails:

git clone --recursive [email protected]:flecsi/flecsi.git
Initialized empty Git repository in /local/home/charest/flecsi/.git/
Warning: Permanently added the RSA host key for IP address '192.30.252.128' to the list of known hosts.
remote: Counting objects: 3722, done.
remote: Compressing objects: 100% (112/112), done.
remote: Total 3722 (delta 57), reused 0 (delta 0), pack-reused 3610
Receiving objects: 100% (3722/3722), 1.24 MiB, done.
Resolving deltas: 100% (2568/2568), done.
Submodule 'cinch' ([email protected]:losalamos/cinch.git) registered for path 'cinch'
Initialized empty Git repository in /local/home/charest/flecsi/cinch/.git/
remote: Counting objects: 1426, done.
remote: Compressing objects: 100% (536/536), done.
remote: Total 1426 (delta 848), reused 1420 (delta 842), pack-reused 0
Receiving objects: 100% (1426/1426), 1.22 MiB | 1.53 MiB/s, done.
Resolving deltas: 100% (848/848), done.
Submodule path 'cinch': checked out '9798489475ba49cdd38eb91663fa0e6044ae11ce'
Submodule 'gtest' ([email protected]:losalamos/google/googletest.git) registered for path 'gtest'
Initialized empty Git repository in /local/home/charest/flecsi/cinch/gtest/.git/
fatal: remote error:
losalamos/google/googletest is not a valid repository name
Email [email protected] for help
Clone of '[email protected]:losalamos/google/googletest.git' into submodule path 'gtest' failed
Failed to recurse into submodule path 'cinch

It looks like you have already completed this, but I thought I'd put it in anyway. You can have the pleasure of closing the issue if it's already done... ;-)

This test depends on the particular order of entries of a std::map, which doesn't seem to be consistent across runs. We need a more robust test that doesn't depend on the specific order of the storage mechanism (this is arbitrary).

Implement vertex and cell centered gradients as outlined in Nathaniel's original requirements document, using wedges and corners in burton mesh.

Using -DENABLE_MPI=ON fails:
https://travis-ci.org/junghans/flecsi/jobs/112658215

CMake Error at CMakeLists.txt:599 (add_test):
  add_test given unknown argument:

    mpi



-- Configuring incomplete, errors occurred!

Most third party libraries dependencies are actually dependencies of exodusii, e.g. hdf5, zlib, netcdf.

We currently use static libraries, which make things a bit harder (see laristra/cinch#9), switching to shared libraries will make this much easier.

Development of basic ALE hydro solver using FleCSI.

G++ does not correctly compile drivers using anything other than the default_execution_policy_t type. This is a bug in G++ with variadic templates. Clang++ is able to correctly compile and run. I suggest that people use clang++ for the time being if they need this functionality. This does not affect the normal build.

• Create common iterator interface (use "->" access where possible)
• Add data versioning (Ben - private issue)
• Add new data interface (Ben - private issue)
• Do 3D (Marc)
• Add documentation
• Enforce 80 column width

Write FindFLECSI.cmake and install it in PREFIX/share.

In d8fb457, @ollielo broke Master, https://travis-ci.org/losalamos/flecsi/jobs/122311235#L541

/home/travis/build/losalamos/flecsi/flecsi/specializations/sagittarius/test/sagittarius_mesh_partition.cc:16:25: fatal error: gmock/gmock.h: No such file or directory

This is actually a bug in cinch!

I need to create a test that will check the change in compilation time when size of the container increases

To avoid template disasters, let's print out build time and ensure it is under 10(?) mins.

We should add a legion build to travis! @spad12 will give me some details on how to build legion using cmake and this will be piece of cake.

\cc @ipdemes @tuxfan

Are we going to use ParMetis for distributed graph partition for the mesh? If so, can you add ParMetis in the third_party repo? Or should this be an issue the flecsi-third-party repo?

This is related to #42, but not possible at that time due to travis-ci/apt-package-safelist#2821, travis-ci/apt-package-safelist#2886 and travis-ci/apt-package-safelist#2874.