epicf / ef Goto Github PK

clang-tidy and clang-format ( http://www.labri.fr/perso/fleury/posts/programming/using-clang-tidy-and-clang-format.html ) seem reasonable options

Split such files as https://github.com/epicf/ef/blob/master/particle_source.cpp and https://github.com/epicf/ef/blob/master/particle_source.h into several files corresponding to separate classes.

Пример конфига: https://drive.google.com/drive/folders/1mDwfGlWc9R7fn-3Ff9ljYWQPfbhMmB1M

Time step was shrinked to 5e-10 from 5e-10 to fit round number of cells.
Time save step was shrinked to 5e-10 from 5e-10 to be a multiple of time step.
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Argument out of range
[0]PETSC ERROR: New nonzero at (8018,8019) caused a malloc
Use MatSetOption(A, MAT_NEW_NONZERO_ALLOCATION_ERR, PETSC_FALSE) to turn off this check
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.6.2, Oct, 02, 2015
[0]PETSC ERROR: ../../ef.out on a x86_64-linux-gnu-real named ab-work by ab Sun Apr 22 23:03:52 2018
[0]PETSC ERROR: Configure options --build=x86_64-linux-gnu --prefix=/usr --includedir=${prefix}/include --mandir=${prefix}/share/man --infodir=${prefix}/share/info --sysconfdir=/etc --localstatedir=/var --with-silent-rules=0 --libdir=${prefix}/lib/x86_64-linux-gnu --libexecdir=${prefix}/lib/x86_64-linux-gnu --with-maintainer-mode=0 --with-dependency-tracking=0 --with-debugging=0 --shared-library-extension=_real --with-hypre=1 --with-hypre-dir=/usr --with-clanguage=C++ --with-c-support --with-shared-libraries --useThreads 0 --with-fortran-interfaces=1 --with-mpi-dir=/usr/lib/openmpi --with-blas-lib=-lblas --with-lapack-lib=-llapack --with-blacs=1 --with-blacs-lib="-lblacsCinit-openmpi -lblacs-openmpi" --with-scalapack=1 --with-scalapack-lib=-lscalapack-openmpi --with-mumps=1 --with-mumps-include="[]" --with-mumps-lib="-ldmumps -lzmumps -lsmumps -lcmumps -lmumps_common -lpord" --with-suitesparse=1 --with-suitesparse-include=/usr/include/suitesparse --with-suitesparse-lib="-lumfpack -lamd -lcholmod -lklu" --with-spooles=1 --with-spooles-include=/usr/include/spooles --with-spooles-lib=-lspooles --with-ptscotch=1 --with-ptscotch-include=/usr/include/scotch --with-ptscotch-lib="-lptesmumps -lptscotch -lptscotcherr" --with-fftw=1 --with-fftw-include="[]" --with-fftw-lib="-lfftw3 -lfftw3_mpi" --with-superlu=1 --with-superlu-include=/usr/include/superlu --with-superlu-lib=-lsuperlu --CXX_LINKER_FLAGS=-Wl,--no-as-needed --prefix=/usr/lib/petscdir/3.6.2/x86_64-linux-gnu-real PETSC_DIR=/build/petsc-16ivCo/petsc-3.6.2.dfsg1 --PETSC_ARCH=x86_64-linux-gnu-real CFLAGS="-g -O2 -fstack-protector-strong -Wformat -Werror=format-security" CXXFLAGS="-g -O2 -fstack-protector-strong -Wformat -Werror=format-security" FCFLAGS="-g -O2 -fstack-protector-strong" FFLAGS="-g -O2 -fstack-protector-strong" CPPFLAGS="-Wdate-time -D_FORTIFY_SOURCE=2" LDFLAGS="-Wl,-Bsymbolic-functions -Wl,-z,relro" MAKEFLAGS=w
[0]PETSC ERROR: #1 MatSetValues_SeqAIJ() line 485 in /build/petsc-16ivCo/petsc-3.6.2.dfsg1/src/mat/impls/aij/seq/aij.c
[0]PETSC ERROR: #2 MatSetValues() line 1173 in /build/petsc-16ivCo/petsc-3.6.2.dfsg1/src/mat/interface/matrix.c
[0]PETSC ERROR: #3 modify_equation_near_object_boundaries() line 216 in field_solver.cpp
terminate called after throwing an instance of 'std::runtime_error'
what(): Error detected in C PETSc
[ab-work:29636] *** Process received signal ***
[ab-work:29636] Signal: Aborted (6)
[ab-work:29636] Signal code: (-6)
[ab-work:29636] [ 0] /usr/lib/libopen-pal.so.13(+0x4e99e)[0x7fbf92a5399e]
[ab-work:29636] [ 1] /lib/x86_64-linux-gnu/libc.so.6(+0x354b0)[0x7fbf99bb84b0]
[ab-work:29636] [ 2] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x38)[0x7fbf99bb8428]
[ab-work:29636] [ 3] /lib/x86_64-linux-gnu/libc.so.6(abort+0x16a)[0x7fbf99bba02a]
[ab-work:29636] [ 4] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(_ZN9__gnu_cxx27__verbose_terminate_handlerEv+0x16d)[0x7fbf9a1f284d]
[ab-work:29636] [ 5] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x8d6b6)[0x7fbf9a1f06b6]
[ab-work:29636] [ 6] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x8d701)[0x7fbf9a1f0701]
[ab-work:29636] [ 7] /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x8d919)[0x7fbf9a1f0919]
[ab-work:29636] [ 8] /usr/lib/x86_64-linux-gnu/libpetsc_real.so.3.6(PetscError+0x351)[0x7fbf9b45a3a1]
[ab-work:29636] [ 9] ../../ef.out[0x429fca]
[ab-work:29636] [10] ../../ef.out[0x42aaa8]
[ab-work:29636] [11] ../../ef.out[0x42ade5]
[ab-work:29636] [12] ../../ef.out[0x4cb7f7]
[ab-work:29636] [13] ../../ef.out[0x47a6eb]
[ab-work:29636] [14] ../../ef.out[0x412a92]
[ab-work:29636] [15] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7fbf99ba3830]
[ab-work:29636] [16] ../../ef.out[0x413099]
[ab-work:29636] *** End of error message ***
Aborted (core dumped)

Single GPU should be sufficient for a start.
CUDA can be used.

A general idea is to define pointers in each class that would hold data about that class on GPU, e.g.

class Time_grid {
  public:
    double total_time, current_time;
    double time_step_size;
    double time_save_step;
    double *total_time_gpu, *current_time_gpu;
    double *time_step_size_gpu;
    double *time_step_step_gpu;
   .....

Then define methods that would allocate memory on GPU and transfer data to and from GPU

class Time_grid {
 .....
  public:
    .....
    void allocate_on_gpu();
    void transfer_to_gpu();
    void transfer_from_gpu();
    ......

Then duplicate all computational methods to perform on GPU. Those methods are supposed to call CUDA kernels.

class Time_grid {
 .....
  public:
  .....
  void update_to_next_step();
  void update_to_next_time_step_gpu();
  .....

Finally, it should be necessary to replace all calls to update methods to calls to update_on_gpu methods and perform data transfer from GPU to CPU when saving on disk is required

void Domain::run_pic()
{
    .....
    for ( int i = current_node; i < total_time_iterations; i++ ){
        std::cout << "Time step from " << i << " to " << i+1
                  << " of " << total_time_iterations << std::endl;
        advance_one_time_step_on_gpu();
        transfer_from_gpu_and_write_step_to_save();
    }
.....
}

Most of the program is computation. Apart from the build system, two other things that would require adjustments are interaction with filesystem and command line. Maybe there is something else.

E.g.
spatial_mesh.cpp -> SpatialMesh.cpp
and
Spatial_mesh -> SpatialMesh

See error and discsussion at
#18 (comment)

Потенциал сферы стал хуже считаться. Это из-за перехода к другому способу расчета поля??

Port some tests from python version: epicf/ef_python#23

See #18 (comment)
Replace with https://stackoverflow.com/questions/13443968/efficient-method-to-check-for-matrix-stability-in-cuda

Currently the program reads ini-config files and stores results in intermediate classes (see https://github.com/epicf/ef/blob/master/config.h and https://github.com/epicf/ef/blob/master/config.cpp ).
Objects of *config* classes are then used to initialize classes that perform computations.

There is no need for such division, so it would be better to remove *config* classes and init computational classes straight from ini-reader.

Calculated results are different from the results shown in Wiki:

After entering of:
git clone https://github.com/epicf/ef your-ef-dir
The terminal answers:
Command 'git' not found but can be installed with:
sudo apt install git

Either adjust or make sure that number of nodes specified by user fits GPU nodes.
See #18 (comment)

Currently there are a lot of classes of ParticleSources and InnerRegions that differ only by underlying geometic shape, e.g: Inner_region_box, Inner_region_cylinder, Particle_source_box, Particle_source_cylinder etc.
It seems more convenient to extract geometric primitives into separate classes and use objects of such classes inside Particle_source and Inner_region.
See also epicf/ef_python#21

Regarding external libraries, small but annoying problems are constantly popping up.
A typical example is build or runtime problem due to library version mismatch on different environments.
Another typical problem is absence of certain libraries on HPC clusters.

For these reasons, it seems desirable to distribute all the necessary libraries with the source code.
Currently, not many libraries are used:

• hdf5lib to save/load results
• boost-pointer-tree to read ini config files
• boost-command-line-options to parse command line
• boost-multiarray to simplify manipulations with 3d arrays
• boost-ptr-vector to simplify work with arrays of objects in inheritance hierarchy
• tinyexpr to define analytical expressions for external fields

hdf5lib is distributed under BSD license. It is necessary to include it in the code and configure compilation. boost-pointer-tree and boost-command-line-options can be replaced with something simpler. E.g. something from https://github.com/nothings/single_file_libs . boost-multiarray can be extracted from boost; alternatively, write some class to handle 3d arrays from scratch. tinyexpr is already included in the repo.