ornl-fusion / korc Goto Github PK

View Code? Open in Web Editor NEW

4.0 11.0 2.0 251.21 MB

Kinetic Orbit Runaway Code (KORC)

Home Page: http://ornl-fusion.github.io/KORC

License: MIT License

Fortran 74.49% Shell 0.64% CMake 0.66% C++ 10.57% MATLAB 12.30% C 0.43% Python 0.89% Makefile 0.01%

electrons runaways fusion

korc's Introduction

KORC

the Kinetic Orbit Runaway electrons Code (KORC) follows relativistic electrons in general electric and magnetic fields under the full Lorentz force, collisions, and radiation losses.

Quick-Start

Spack provides an easy way to setup the gfortran compiler toolchain and other dependencies without the headache of managing them through the OS. Here's how to build KORC on a Linux workstation using Spack:

Grab the spack software using git, and source the setup-env.sh script from within the spack installation.

[user@localhost KORC]$ git clone --depth=100 --branch=releases/v0.21 https://github.com/spack/spack.git ~/spack
[user@localhost KORC]$ . ~/spack/share/spack/setup-env.sh

Create and activate a new spack environment

[KORC] [user@localhost KORC]$ spack env create -d .
[KORC] [user@localhost KORC]$ spack env activate -p -d .

Configure spack and bootstrap the compiler

[KORC] [user@localhost KORC]$ spack config add "config:install_tree:padded_length:128"

[KORC] [user@localhost KORC]$ spack install --no-cache --add [email protected]
[KORC] [user@localhost KORC]$ spack load [email protected]
[KORC] [user@localhost KORC]$ spack compiler find

[KORC] [user@localhost KORC]$ spack install --no-cache --add cmake %[email protected]
[KORC] [user@localhost KORC]$ spack load cmake %[email protected]

[KORC] [user@localhost KORC]$ spack concretize -f
[KORC] [user@localhost KORC]$ spack install --no-cache --add hdf5+fortran+mpi %[email protected]
[KORC] [user@localhost KORC]$ spack load hdf5+fortran+mpi %[email protected]

Verify gfortan, cmake and Build KORC

[KORC] [user@localhost KORC]$ which cmake
[KORC] [user@localhost KORC]$ which gfortran
[KORC] [user@localhost KORC]$ gfortran --version

[KORC] [user@localhost KORC]$ ./build.sh

Exit the spack environment

[KORC] [user@localhost KORC]$ spack env deactivate

More Info

http://ornl-fusion.github.io/KORC

korc's People

Contributors

Stargazers

Watchers

Forkers

fusion-impurity idigs

korc's Issues

Set a default build type

Set a default build type like we do in ASGarD

https://www.kitware.com/cmake-and-the-default-build-type/
https://github.com/openchemistry/tomviz/blob/master/cmake/BuildType.cmake

ExCL self-hosted github runner

Replicate the Human-in-loop CI using ideas from https://github.com/JuliaORNL/JACC.jl/blob/main/.github/workflows/ci-gpu-AMD.yaml
and implement for KORC

setup spack env file

Translate the imperative spack commands in the action workflow into a spack env yml file

CMake build system

I added a cmake build system to KORC, this is a start and we can build from here.

added CMakeLists.txt, the current code sits in a sub-directory, we can add new sub-directories as we convert the code to C++
added FindPspline module, the pspline folder should be specified with either CMAKE_PREFIX_PATH or Pspline_ROOT_DIR
added option for double vs single precision, KORC_ENABLE_DOUBLE_PRECISION is ON by default

@lcarbajal, check the code, we can merge into master, if you are happy with the changes.

testing readme

Write how we want to setup testing in CI/CD

spack build on perlmutter

https://nersc-spack-infrastructure.readthedocs.io/en/latest/spack_training.html

pin spack workflow builds to concretized spec

Marry the concretized spec with spack cache

KORC random uses global state variables.

The Problem

The korc_random module uses global variables for the random state objects. This can cause issues with nest calls to the allocation and deallocation routines.

subroutine foo
    real (rp) :: value
    call initialize_random(0)
    call bar
    value = get_random()
    call finalize_random
end subroutine

subroutine bar
    call initialize_random(0)
    call finalize_random
end subroutine

This has a number of problems. Bar overwrites and destroys the state objects before foo can use them.

Suggested solution

Wrap the random functions in a fortran 2003 class object.

TYPE :: random_state
    TYPE(C_PTR) :: state
CONTAINS
    PROCEDURE :: get => random_state_get
    FINAL     :: random_state_destruct
END TYPE

FUNCTION random_state_construct(seed)
    CLASS (random_state), POINTER :: random_state_construct
    INTEGER, INTENT(in)           :: seed
    ALLOCATE(random_state_construct)
    random_state_construct%state = random_construct_U(seed)
END FUNCTION

SUBROUTINE random_state_destruct(this)
    TYPE (random_state), INTENT(inout) :: this

    CALL random_destroy_U(this%state)
END SUBROUTINE

FUNCTION random_state_get(this)
    REAL (rp)                          :: random_state_get
    TYPE (random_state), INTENT(inout) :: this
    random_state_get = random_get_number_U(this%state)
END FUNCTION

This would allow

subroutine foo
    CLASS(random_state), POINTER :: r_state
    REAL (rp)                    :: value
    r_state => random_state_construct(0)
    call bar
    value = r_state%get()
    DEALLOCATE (r_state)
end subroutine

subroutine bar
    CLASS(random_state), POINTER :: r_state
    r_state => random_state_construct(0)
    DEALLOCATE (r_state)
end subroutine

memory leaks in egyro_test

Address Sanitizer reports two memory leaks with a full call stack. Let's address these first.

1: Direct leak of 2536 byte(s) in 1 object(s) allocated from:
1:     #0 0x7f67b4b89587 in operator new(unsigned long) ../../../../src/libsanitizer/asan/asan_new_delete.cc:104
1:     #1 0x559e1b7dc34e in random_construct_N /home/svh/Documents/KORC/src/korc_c_random.cpp:50
1:     #2 0x559e1b765e4a in __korc_random_MOD_initialize_random_n /home/svh/Documents/KORC/src/korc_random.f90:142
1:     #3 0x559e1b70ac16 in __korc_rnd_numbers_MOD_init_random_seed /home/svh/Documents/KORC/src/korc_rnd_numbers.f90:163
1:     #4 0x559e1b757616 in gyro_distribution /home/svh/Documents/KORC/src/korc_velocity_distribution.f90:416
1:     #5 0x559e1b754feb in __korc_velocity_distribution_MOD_initial_gyro_distribution /home/svh/Documents/KORC/src/korc_velocity_distribution.f90:506
1:     #6 0x559e1b725350 in __korc_initialize_MOD_set_up_particles_ic /home/svh/Documents/KORC/src/korc_initialize.f90:733
1:     #7 0x559e1b55961f in MAIN__ /home/svh/Documents/KORC/src/main.f90:270
1:     #8 0x559e1b55a706 in main /home/svh/Documents/KORC/src/main.f90:7
1:     #9 0x7f67b3f0a082 in __libc_start_main ../csu/libc-start.c:308

1: Direct leak of 2520 byte(s) in 1 object(s) allocated from:
1:     #0 0x7f67b4b89587 in operator new(unsigned long) ../../../../src/libsanitizer/asan/asan_new_delete.cc:104
1:     #1 0x559e1b7dc2ed in random_construct_U /home/svh/Documents/KORC/src/korc_c_random.cpp:46
1:     #2 0x559e1b765ea4 in __korc_random_MOD_initialize_random_u /home/svh/Documents/KORC/src/korc_random.f90:133
1:     #3 0x559e1b70aba9 in __korc_rnd_numbers_MOD_init_random_seed /home/svh/Documents/KORC/src/korc_rnd_numbers.f90:162
1:     #4 0x559e1b5c04a8 in __korc_fields_MOD_unitvectors /home/svh/Documents/KORC/src/korc_fields.f90:1101
1:     #5 0x559e1b758934 in gyro_distribution /home/svh/Documents/KORC/src/korc_velocity_distribution.f90:437
1:     #6 0x559e1b754feb in __korc_velocity_distribution_MOD_initial_gyro_distribution /home/svh/Documents/KORC/src/korc_velocity_distribution.f90:506
1:     #7 0x559e1b725350 in __korc_initialize_MOD_set_up_particles_ic /home/svh/Documents/KORC/src/korc_initialize.f90:733
1:     #8 0x559e1b55961f in MAIN__ /home/svh/Documents/KORC/src/main.f90:270
1:     #9 0x559e1b55a706 in main /home/svh/Documents/KORC/src/main.f90:7
1:     #10 0x7f67b3f0a082 in __libc_start_main ../csu/libc-start.c:308

investigate github dependency cache invalidation

We need a way to elect cache invalidation for our spack mirror and spack.lock concretized spec

spack garbage collect when building github cache

Synthetic Camera OpenMP has race conditions.

There are multiple problems spotted in the OpenMP parallelization of the synthetic camera. Also factor out redundant code to a common subroutine. This should simplify a bunch of the OpenMP directives.

Disable in-source builds

Our GitHub actions CI is currently executing in-source builds. Let's fix this, the build/build directory out-of-source builds currenlty have and add a CMake check to guard against this.

https://dpiepgrass.medium.com/cmake-fights-against-disabling-in-source-builds-ab1d71c1d26f

Found in #10.

Pchunk is depricated

Transitioning to GPU platforms for large-scale simulations has made SIMD vectorization unnecessary. PSPLINE routines are now serial, and routines called from korc_ppusher must be made serial. This also alleviates the issue of constantly allocating small vector arrays.

KORC fails to build with latest `master`

I am no longer able to build KORC with gfortran-9 or gfortran-11 at commit 38ed2f3 (current master). The error message is below

/home/svh/Documents/KORC/src/korc_ppusher.f90:19:34:

   19 |   TYPE(KORC_2D_FIELDS_INTERPOLANT)      :: bfield_2d_local
      |                                  1
Error: Derived type ‘korc_2d_fields_interpolant’ at (1) is being used before it is defined
/home/svh/Documents/KORC/src/korc_ppusher.f90:20:34:

   20 |   TYPE(KORC_2D_FIELDS_INTERPOLANT)      :: b1Refield_2d_local
      |                                  1
Error: Derived type ‘korc_2d_fields_interpolant’ at (1) is being used before it is defined
/home/svh/Documents/KORC/src/korc_ppusher.f90:21:34:

   21 |   TYPE(KORC_2D_FIELDS_INTERPOLANT)      :: b1Imfield_2d_local
      |                                  1
Error: Derived type ‘korc_2d_fields_interpolant’ at (1) is being used before it is defined
/home/svh/Documents/KORC/src/korc_ppusher.f90:22:31:

   22 |   TYPE(KORC_INTERPOLANT_DOMAIN)        :: fields_domain_local
      |                               1
Error: Derived type ‘korc_interpolant_domain’ at (1) is being used before it is defined
/home/svh/Documents/KORC/src/korc_ppusher.f90:23:35:

   23 |   TYPE(KORC_2DX_FIELDS_INTERPOLANT)      :: b1Refield_2dx_local
      |                                   1
Error: Derived type ‘korc_2dx_fields_interpolant’ at (1) is being used before it is defined
/home/svh/Documents/KORC/src/korc_ppusher.f90:24:35:

   24 |   TYPE(KORC_2DX_FIELDS_INTERPOLANT)      :: b1Imfield_2dx_local
      |                                   1
Error: Derived type ‘korc_2dx_fields_interpolant’ at (1) is being used before it is defined
/home/svh/Documents/KORC/src/korc_ppusher.f90:25:35:

   25 |   TYPE(KORC_2DX_FIELDS_INTERPOLANT)      :: e1Refield_2dx_local
      |                                   1
Error: Derived type ‘korc_2dx_fields_interpolant’ at (1) is being used before it is defined
/home/svh/Documents/KORC/src/korc_ppusher.f90:26:35:

   26 |   TYPE(KORC_2DX_FIELDS_INTERPOLANT)      :: e1Imfield_2dx_local
      |                                   1
Error: Derived type ‘korc_2dx_fields_interpolant’ at (1) is being used before it is defined
/home/svh/Documents/KORC/src/korc_ppusher.f90:935:93:

  935 |       call provide_ezspline_aorsa_ACC(bfield_2d_local,b1Refield_2dx_local,b1Imfield_2dx_local, &
      |                                                                                             1
Error: Symbol ‘b1imfield_2dx_local’ at (1) has no IMPLICIT type
/home/svh/Documents/KORC/src/korc_ppusher.f90:935:73:

  935 |       call provide_ezspline_aorsa_ACC(bfield_2d_local,b1Refield_2dx_local,b1Imfield_2dx_local, &
      |                                                                         1
Error: Symbol ‘b1refield_2dx_local’ at (1) has no IMPLICIT type
/home/svh/Documents/KORC/src/korc_ppusher.f90:935:53:

  935 |       call provide_ezspline_aorsa_ACC(bfield_2d_local,b1Refield_2dx_local,b1Imfield_2dx_local, &
      |                                                     1
Error: Symbol ‘bfield_2d_local’ at (1) has no IMPLICIT type
/home/svh/Documents/KORC/src/korc_ppusher.f90:936:47:

  936 |         e1Refield_2dx_local,e1Imfield_2dx_local,fields_domain_local)
      |                                               1
Error: Symbol ‘e1imfield_2dx_local’ at (1) has no IMPLICIT type
/home/svh/Documents/KORC/src/korc_ppusher.f90:936:27:

  936 |         e1Refield_2dx_local,e1Imfield_2dx_local,fields_domain_local)
      |                           1
Error: Symbol ‘e1refield_2dx_local’ at (1) has no IMPLICIT type
/home/svh/Documents/KORC/src/korc_ppusher.f90:936:67:

  936 |         e1Refield_2dx_local,e1Imfield_2dx_local,fields_domain_local)
      |                                                                   1
Error: Symbol ‘fields_domain_local’ at (1) has no IMPLICIT type
/home/svh/Documents/KORC/src/korc_ppusher.f90:716:90:

  716 |       call provide_ezspline_mars_ACC(bfield_2d_local,b1Refield_2d_local,b1Imfield_2d_local, &
      |                                                                                          1
Error: Symbol ‘b1imfield_2d_local’ at (1) has no IMPLICIT type
/home/svh/Documents/KORC/src/korc_ppusher.f90:716:71:

  716 |       call provide_ezspline_mars_ACC(bfield_2d_local,b1Refield_2d_local,b1Imfield_2d_local, &
      |                                                                       1
Error: Symbol ‘b1refield_2d_local’ at (1) has no IMPLICIT type
/home/svh/Documents/KORC/src/korc_ppusher.f90:716:52:

  716 |       call provide_ezspline_mars_ACC(bfield_2d_local,b1Refield_2d_local,b1Imfield_2d_local, &
      |                                                    1
Error: Symbol ‘bfield_2d_local’ at (1) has no IMPLICIT type
/home/svh/Documents/KORC/src/korc_ppusher.f90:717:27:

  717 |         fields_domain_local)
      |                           1
Error: Symbol ‘fields_domain_local’ at (1) has no IMPLICIT type
make[2]: *** [src/CMakeFiles/xkorc.dir/build.make:153: src/CMakeFiles/xkorc.dir/korc_ppusher.f90.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:98: src/CMakeFiles/xkorc.dir/all] Error 2
make: *** [Makefile:91: all] Error 2
svh@LAP131024:~/Documents/KORC/build$