Giter Club home page Giter Club logo

impactx's Introduction

ImpactX

CI Status Documentation Status License ImpactX Supported Platforms
DOI (source) DOI (paper)
Language: C++17 Language: Python

ImpactX: an s-based beam dynamics code including space charge effects. This is the next generation of the IMPACT-Z code.

Documentation

In order to learn how to install and run the code, please see the online documentation: https://impactx.readthedocs.io

Contributing

AMReX

Our workflow is described in CONTRIBUTING.rst.

Developer Environment

Please see our developer installation section of the documentation for an easy install of our software dependencies.

Get the Source Code

Before you start, you will need a copy of the ImpactX source code:

git clone [email protected]:ECP-WarpX/impactx.git
cd impactx

Compile

# find dependencies & configure
cmake -S . -B build

# compile
cmake --build build -j 4

That's all! ImpactX binaries are now in build/bin/. Most people execute these binaries directly or copy them out.

You can inspect and modify build options after running cmake -S . -B build with either

ccmake build

or by adding arguments with -D<OPTION>=<VALUE> to the first CMake call, e.g.:

cmake -S . -B build -DImpactX_COMPUTE=CUDA -DImpactX_MPI=OFF

Python Compile

# find dependencies & configure
cmake -S . -B build -DImpactX_PYTHON=ON

# compile & install
cmake --build build -j 4 --target pip_install

Run

An executable ImpactX binary with the current compile-time options encoded in its file name will be created in build/bin/.

Additionally, a symbolic link named impactx can be found in that directory, which points to the last built ImpactX executable.

The command-line syntax for this executable is:

Usage: impactx <inputs-file> [some.overwritten.option=value]...

Mandatory arguments (remove the <>):
  inputs-file     the path to an input file; can be relative to the current
                  working directory or absolute.
                  Example: input_fodo.in

Optional arguments (remove the []):
  options         this can overwrite any line in an inputs-file
                  Example: quad1.ds=0.5 sbend1.rc=1.5

Examples:
  In the current working directory, there is a file "input_fodo.in" and the
  "impactx" executable.
  The line to execute would look like this:
    ./impactx input_fodo.in

  In the current working directory, there is a file "input_fodo.in" and the
  executable "impactx" is in a directory that is listed in the "PATH"
  environment variable.
  The line to execute would look like this:
    impactx input_fodo.in

  In the current working directory, there is a file "input_fodo.in" and the
  "impactx" executable. We want to voerwrite the segment length of the beamline
  element "quad1" that is already defined in it. We also want to change the
  radius of curvature of the bending magnet "sbend1" to a different value than
  in the file "input_fodo.in".
  The line to execute would look like this:
    ./impactx input_fodo.in quad1.ds=0.5 sbend1.rc=1.5

Test

In order to run our tests, you need to have a few Python packages installed:

python3 -m pip install --upgrade pip
python3 -m pip install --upgrade build packaging setuptools wheel pytest
python3 -m pip install --upgrade -r tests/python/requirements.txt

You can run all our tests with:

ctest --test-dir build --output-on-failure

Further options:

  • help: ctest --test-dir build --help
  • list all tests: ctest --test-dir build -N
  • only run tests that have "FODO" in their name: ctest --test-dir build -R FODO

Acknowledgements

This work was supported by the Laboratory Directed Research and Development Program of Lawrence Berkeley National Laboratory under U.S. Department of Energy Contract No. DE-AC02-05CH11231.

ImpactX is supported by the CAMPA collaboration, a project of the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research and Office of High Energy Physics, Scientific Discovery through Advanced Computing (SciDAC) program.

Copyright Notice

ImpactX Copyright (c) 2022, The Regents of the University of California, through Lawrence Berkeley National Laboratory (subject to receipt of any required approvals from the U.S. Dept. of Energy). All rights reserved.

If you have questions about your rights to use or distribute this software, please contact Berkeley Lab's Intellectual Property Office at [email protected].

Please see the full license agreement in LICENSE.txt.
Please see the notices in NOTICE.txt.
The SPDX license identifier is BSD-3-Clause-LBNL.

impactx's People

Contributors

atmyers avatar ax3l avatar cemitch99 avatar dependabot[bot] avatar n01r avatar pre-commit-ci[bot] avatar qianglbl avatar remilehe avatar rtsandberg avatar s-sajid-ali avatar tanweihou avatar weiqunzhang avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

impactx's Issues

Activate Dynamic Load Balancing

Start like this:

  • expose a user parameter with N boxes per device; currently, N is hard-coded to 1
  • add a cost function
  • add load balance interval control

Conda: llvm-strip <=13 broken on Apple aarch64 M1

Cross-Link of AMReX-Codes/pyamrex#35

If you compile with the Conda-Forge compilers on Apple M1 processors, you need to add

cmake -S . -B build -DImpactX_PYTHON=ON -DCMAKE_STRIP=$(which strip)

to avoid errors of the kind:

zsh: killed     python -c "import impactx"

This is because the llvm-strip command is not compatible to the XCode strip command yet, which we use to reduce the pyAMReX binary size significantly.

LLVM 14 should fix this (see above linked issue for updates).

Default (Top) Visualization of a Lattice

It would be super useful if we could do a simple z-x visualization of an accelerator lattice, e.g., from the Python side.

We could write a function that takes our list of Python elements and saves them to SVG/PDF/PNG/TiX, etc. I am not yet sure if matplotlib or another tool would be ideal for this job.

Illegal memory access with FODO example on GPU

Hi, I tried to run the FODO example without changes on the Perlmutter GPU partition and encountered the following error:

amrex::Abort::0::CUDA error 700 in file /global/homes/m/mgarten/src/impactx/build/_deps/fetchedamrex-src/Src/Base/AMReX_GpuDevice.cpp line 660: an illegal memory access was encountered !!!
SIGABRT
See Backtrace.0 file for details
MPICH ERROR [Rank 0] [job id 2519166.0] [Wed Jun 29 18:07:42 2022] [nid001512] - Abort(6) (rank 0 in comm 480): application called MPI_Abort(comm=0x84000001, 6) - process 0

srun: error: nid001512: task 0: Exited with exit code 6
srun: launch/slurm: _step_signal: Terminating StepId=2519166.

I first tried the example with the submit script that is provided by the docs (however, had to change the naming a bit since it's still copied from WarpX).
This configuration used 4 nodes but I also tried a single node and then just a single GPU per node. All fail with the same error.

The backtrace reads the following:

Backtrace.0
=== If no file names and line numbers are shown below, one can run
            addr2line -Cpfie my_exefile my_line_address
    to convert `my_line_address` (e.g., 0x4a6b) into file name and line number.
    Or one can use amrex/Tools/Backtrace/parse_bt.py.

=== Please note that the line number reported by addr2line may not be accurate.
    One can use
            readelf -wl my_exefile | grep my_line_address'
    to find out the offset for that line.

 0: /pscratch/sd/m/mgarten/impactx/001_FODO_single-GPU/./impactx() [0x5d63b6]
    amrex::BLBackTrace::print_backtrace_info(_IO_FILE*) at ??:?

 1: /pscratch/sd/m/mgarten/impactx/001_FODO_single-GPU/./impactx() [0x5d875c]
    amrex::BLBackTrace::handler(int) at ??:?

 2: /pscratch/sd/m/mgarten/impactx/001_FODO_single-GPU/./impactx() [0x5c13e9]
    amrex::Gpu::Device::streamSynchronizeAll() at ??:?

 3: /pscratch/sd/m/mgarten/impactx/001_FODO_single-GPU/./impactx() [0x5b6165]
    amrex::MFIter::~MFIter() at ??:?

 4: /pscratch/sd/m/mgarten/impactx/001_FODO_single-GPU/./impactx() [0x473e79]
    impactx::Push(impactx::ImpactXParticleContainer&, std::__cxx11::list<std::variant<impactx::None, impactx::ConstF, impactx::DipEdge, impactx::Drift, impactx::Multipole, impactx::NonlinearLens, impactx::Quad, impactx::Sbend, impactx::ShortRF>, std::allocator<std::variant<impactx::None, impactx::ConstF, impactx::DipEdge, impactx::Drift, impactx::Multipole, impactx::NonlinearLens, impactx::Quad, impactx::Sbend, impactx::ShortRF> > > const&) at ??:?

 5: /pscratch/sd/m/mgarten/impactx/001_FODO_single-GPU/./impactx() [0x432b06]
    impactx::ImpactX::evolve(int) at ??:?

 6: /pscratch/sd/m/mgarten/impactx/001_FODO_single-GPU/./impactx() [0x41ecfe]
    main at ??:?

 7: /lib64/libc.so.6(__libc_start_main+0xef) [0x7f2ab60502bd]

 8: /pscratch/sd/m/mgarten/impactx/001_FODO_single-GPU/./impactx() [0x42da0a]
    _start
../sysdeps/x86_64/start.S:122


===== TinyProfilers ======
main()
ImpactX::evolve
ImpactX::evolve::step
"Backtrace.0" 43L, 1979C                                                                                                                                                            1,66          Top

I am adding the machine/system label since an earlier Slack message from @ax3l said that it runs on Summit V100s without fail.

Diagnostics as Zero-Sized Elements

Placing diagnostics properly in evolve-loops is always a hazzle that complicates the core logic of PIC codes (before/after the loop, what is step zero, diagnostics at a fixed position s that might not coincide with the end of an element, etc.).

With our lattice elements as main outer loop of the code, we could do the following approach: we could treat at diagnostics as zero-width element and just put it in the lattice elements list. That way, order is 100% clear.

This also has a nice analogy to actual, physical beam monitors.

As syntactic sugar for users, we could still provide input options that let them set the diagnostics at a given reference s (or time/phase) and then we find the corresponding element that might be cut and slice it up automatically into two at the location. If the position fits the end of an element, then no slicing is needed.

Implement unused parameter warnings (& optional abort)

As in WarpX, we want to add warnings after step 1 (and at the end) about unused parameters from inputs files.
This helps users to spot typos and logic issues quickly.

We also need an "abort on warnings" options for our CI (unsupervised, automated tests), so that our inputs files always work.

First seen in #118

MAD-X input reader: constants, elements, lines, segments

We want to support reading accelerator lattices directly from MAD-X input files.
For this, we will need to support at least the following syntax elements:

  • constants
  • elements
  • lines
  • segments
  • simple math

from MAD-X definitions.

For that, we could explore if @janfschmidt's palattice could be used as a library :)
https://github.com/janfschmidt/palattice

References on MAD-X:

Example files for MAD-X lattices:

In-Situ Beam Parameter Calculation for Reduced Diagnostics

Adding in-situ calculated beam parameters to ImpactX would reduce I/O and make typically used quantities like beta-function, emittance and distribution momenta readily available without additional post-processing.

  • Implement beam parameter calculation
    • sigma_x,y,z
    • <x/y/z>
    • <p_x/y/z>
    • min/max
    • emittance_x/y/z
    • alpha: x'-... (only for fully matched beam equal to twiss paramters)
  • add ReducedDiagnostics for them
  • user-facing docs
  • implement/update tests

OMP CTest runs fail on MacOS

When running ctest on the release build of ImpactX (30f93ec, AMReX 23.04) on a MacBook Pro 35 tests out of 94 are failing after the run and TinyProfiler output, after AMReX announces that it finalizes.

E.g. here the error from FODO.run

Pinned Memory Usage:
------------------------------------------------------------------
Name                             Nalloc  Nfree    AvgMem    MaxMem
------------------------------------------------------------------
The_Pinned_Arena::Initialize()        1      1    95 KiB  8192 KiB
ImpactX::AddNParticles               27     27  2524   B   853 KiB
ParticleContainer::addParticles     846    846   400 KiB   853 KiB
------------------------------------------------------------------

AMReX (23.04) finalized
[MacBookPro:16898] *** Process received signal ***
[MacBookPro:16898] Signal: Segmentation fault: 11 (11)
[MacBookPro:16898] Signal code: Address not mapped (1)
[MacBookPro:16898] Failing at address: 0x10
[MacBookPro:16898] [ 0] 0   libsystem_platform.dylib            0x00007ff80e4055ed _sigtramp + 29
[MacBookPro:16898] [ 1] 0   ???                                 0x0000000000000021 0x0 + 33
[MacBookPro:16898] [ 2] 0   libomp.dylib                        0x000000010ff74342 __kmp_release_64 + 34
[MacBookPro:16898] [ 3] 0   libomp.dylib                        0x000000010ff4d027 _ZL17__kmp_reap_threadP8kmp_infoi + 471
[MacBookPro:16898] [ 4] 0   libomp.dylib                        0x000000010ff4a399 _ZL18__kmp_internal_endv + 185
[MacBookPro:16898] [ 5] 0   libomp.dylib                        0x000000010ff4a226 __kmp_internal_end_library + 518
[MacBookPro:16898] [ 6] 0   libsystem_c.dylib                   0x00007ff80e2a9ba1 __cxa_finalize_ranges + 409
[MacBookPro:16898] [ 7] 0   libsystem_c.dylib                   0x00007ff80e2a99bb exit + 35
[MacBookPro:16898] [ 8] 0   libdyld.dylib                       0x00007ff80e3ef8d3 _ZNK5dyld416LibSystemHelpers4exitEi + 11
[MacBookPro:16898] [ 9] 0   dyld                                0x00007ff80e07e458 start + 1960
[MacBookPro:16898] *** End of error message ***
<end of output>

MacOS Version 13.3.1 (22E261) (just updated today but also failed with previous version)

I am using Spack for the dependencies. This is the content of spack.yaml:

# This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
spack:
  # add package specs to the `specs` list
  specs: [cmake, fftw, mpi, llvm-openmp, adios2, hdf5, pkgconfig, python, py-pip,
    py-pybind11, py-mpi4py]
  view: true
  concretizer:
    unify: true
  compilers:
  - compiler:
      spec: [email protected]
      paths:
        cc: /usr/bin/clang
        cxx: /usr/bin/clang++
        f77: null
        fc: null
      flags: {}
      operating_system: monterey
      target: x86_64
      modules: []
      environment: {}
      extra_rpaths: []

Space-Charge: AMR

Based on #102, implement adaptive, multi-level mesh-refinement

Decide on refinement criteria, e.g., gradients of charge density.

Plasma Lens Element

In ECP-WarpX/WarpX#3063, @dpgrote adds a simple plasma lens element.

plasmalens for a field modeling a plasma lens
This applies a radially directed plasma lens field that is uniform within the z extent of the element with
a sharp cut off at the ends.
This uses residence corrections, with the field scaled by the amount of time within the element for particles entering
or leaving it, to increase the accuracy.

Let us add the same element as a map to ImpactX.

Is this similar to our ConstF element?

Line Element: Repeat

Add a "repeat" option to Line to make building of channels with many periods easy.

Combined Function Magnet

We received an email request for a combined function magnets (a dipole or “SBEND” with a quadrupole strenght associated with) and will also require such an element to model ALS-U.

MAD-X has this as a a quadrupole field that one can set to a dipole.

After a first discussion, we could implement this as:

  1. linear version
  2. w/ non-linear elements through symplectic integration

Checkpoint Restart Functionality

Add checkpoint-restart functionality to ImpactX.

Needs to restore:

  • simulation,
  • diagnostics

states, incl. written files from in situ diagnostics, etc.

no attribute 'BeamMonitor' in conda install

Hey folks, I was trying to get a test example running (specifically the Constant Focusing Channel example). I am able to successfully install the package through conda. However, executing the script gives the following error.

Traceback (most recent call last):
  File "/home/chris/physics-ml-priors-linux/20230402-impactx-test.py", line 41, in <module>
    monitor = elements.BeamMonitor("monitor", "h5")
AttributeError: module 'impactx.impactx_pybind.elements' has no attribute 'BeamMonitor'

Handling of Single Particles / Perfectly Flat / Line Beams

While running in debug (#43), I realized the ResizeMesh function does assert the simulation:
https://github.com/ECP-WarpX/impactx/pull/19/files#r778820245

0::Assertion `hi > lo' failed, file "/home/axel/src/impactx/build/_deps/fetchedamrex-src/Src/Base/AMReX_Algorithm.H", line 107, Msg: "Error - calling bisect but lo and hi don't describe a reasonable interval." !!!

I temporarily disabled calling it and enabled debug builds in CI (#43). The function probably needs to be fixed.

Reference Particle Helpers

Let's add helper functions to the reference particle to:

  • set energy in MeV
  • get energy in MeV, gamma, gamma-1
  • get momentum in normalized units, beta*gamma
  • get beta

Profiling ImpactX on Crusher with AMD tools

During the OLCF crusher hackathon I had tried to build/run ImpactX for profiling.
I tried a CPU-only version first to then go ahead and use AMD-provided tools on Crusher for profiling of the host side.
However, running failed and I am trying again today to document the status of where it is failing.

To reproduce, first load modules for the dependencies (starting off here from the one for Crusher from the WarpX repository since ImpactX and WarpX share the same dependencies).
https://warpx.readthedocs.io/en/latest/install/hpc/crusher.html#id1
For convenience, download and save under $HOME/crusher_impactx.profile.

Commands used to build the most recent development branch

source $HOME/crusher_impactx.profile

mkdir -p $HOME/src
# download defaults to development branch
git clone https://github.com/ECP-WarpX/impactx.git $HOME/src/impactx
cd $HOME/src/impactx

cmake -S . -B build -DImpactX_COMPUTE=NOACC -DCMAKE_BUILD_TYPE=RelWithDebInfo
cmake --build build -j 16

Running the FODO example works with this.
However, using Omnitrace on this example did not (back during last year)

  • follow up with more details on that.

On the other hand, building for HIP did not work and the install failed

rm -rf build

cmake -S . -B build -DImpactX_COMPUTE=HIP -DCMAKE_BUILD_TYPE=RelWithDebInfo
cmake --build build -j 16

See the build and error output by expanding the following details section

[[email protected] impactx]$ cmake -S . -B build -DImpactX_COMPUTE=NOACC -DCMAKE_BUILD_TYPE=RelWithDebInfo
-- Cray Programming Environment 2.7.17 C
-- Cray Programming Environment 2.7.17 CXX
-- Found CCache: /sw/crusher/spack-envs/base/opt/linux-sles15-x86_64/gcc-7.5.0/ccache-4.5.1-gq5xkxxgeqkxopbjy6gacidd36xglhqc/bin/ccache
-- Downloading ABLASTR ...
-- ABLASTR repository: https://github.com/ECP-WarpX/WarpX.git (23.01)
-- Found CCache: /sw/crusher/spack-envs/base/opt/linux-sles15-x86_64/gcc-7.5.0/ccache-4.5.1-gq5xkxxgeqkxopbjy6gacidd36xglhqc/bin/ccache
-- Downloading AMReX ...
-- AMReX repository: https://github.com/AMReX-Codes/amrex.git (23.01)
-- CMake version: 3.23.2
-- AMReX installation directory: /usr/local
-- Build type set by user to 'RelWithDebInfo'.
-- Building AMReX with AMReX_SPACEDIM = 3
-- Configuring AMReX with the following options enabled: 
--    AMReX_BUILD_SHARED_LIBS
--    AMReX_PRECISION = DOUBLE
--    AMReX_MPI
--    AMReX_MPI_THREAD_MULTIPLE
--    AMReX_LINEAR_SOLVERS
--    AMReX_PARTICLES
--    AMReX_PARTICLES_PRECISION = DOUBLE
--    AMReX_PIC
--    AMReX_TINY_PROFILE
-- AMReX configuration summary: 
--    Build type               = RelWithDebInfo
--    Install directory        = /usr/local
--    C++ compiler             = /opt/cray/pe/craype/2.7.17/bin/CC
--    C++ defines              = 
--    C++ flags                = -O2 -g -DNDEBUG -I/opt/rocm-5.2.0/include 
--    C++ include paths        = -I/ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-src/Src/Base -I/ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-src/Src/Base/Parser -I/ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-src/Src/Boundary -I/ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-src/Src/AmrCore -I/ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-src/Src/LinearSolvers/MLMG -I/ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-src/Src/LinearSolvers/OpenBC -I/ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-src/Src/Particle
--    Link line                = 
-- AMReX: Using version '23.01' (23.01)

WarpX build configuration:
  Version: 23.01 (23.01)
  C++ Compiler: Clang 14.0.6 CrayPrgEnv
    /opt/cray/pe/craype/2.7.17/bin/CC

  Installation prefix: /usr/local
        bin: bin
        lib: lib64
    include: include
      cmake: lib64/cmake/WarpX

  Build type: RelWithDebInfo
  Build options:
    APP: OFF
    ASCENT: OFF
    COMPUTE: NOACC
    DIMS: 3
    Embedded Boundary: OFF
    GPU clock timers: OFF
    IPO/LTO: OFF
    LIB: OFF
    MPI: ON
    PSATD: OFF
    PRECISION: DOUBLE
    PARTICLE PRECISION: DOUBLE
    OPENPMD: OFF
    QED: OFF
    QED table generation: OFF
    SENSEI: OFF

-- ABLASTR: Using version '' (23.01)
-- Downloading pybind11 ...
-- pybind11 repository: https://github.com/pybind/pybind11.git (v2.10.1)
-- pybind11 v2.10.1 
-- Downloading pyAMReX ...
-- pyAMReX repository: https://github.com/AMReX-Codes/pyamrex.git (f3ba017531e2d1569e63428919632cf908435e56)
-- AMReX::amrex target already imported
-- pybind11::module target already imported

pyAMReX build configuration:
  Version: 23.01
  C++ Compiler: Clang 14.0.6 CrayPrgEnv
    /opt/cray/pe/craype/2.7.17/bin/CC

  Installation prefix: /usr/local
        bin: bin
        lib: lib64
    include: include
      cmake: lib64/cmake/pyAMReX
     python: lib64/python3.9/site-packages

  Build type: RelWithDebInfo


ImpactX build configuration:
  Version: 23.02 (23.02-1-g1a134a03ef79)
  C++ Compiler: Clang 14.0.6 CrayPrgEnv
    /opt/cray/pe/craype/2.7.17/bin/CC

  Installation prefix: /usr/local
        bin: bin
        lib: lib64
    include: include
      cmake: lib64/cmake/ImpactX
     python: lib64/python3.9/site-packages

  Build type: RelWithDebInfo
  Build options:
    APP: ON
    COMPUTE: NOACC
    IPO/LTO: OFF
    LIB: ON (static)
    MPI: ON
    PRECISION: DOUBLE
    PYTHON: ON
    OPENPMD: OFF

-- Configuring done
-- Generating done
-- Build files have been written to: /ccs/home/mgarten/src/impactx/build
[[email protected] impactx]$ rm -rf build
[[email protected] impactx]$ cmake -S . -B build -DImpactX_COMPUTE=HIP -DCMAKE_BUILD_TYPE=RelWithDebInfo
-- The C compiler identification is Clang 14.0.6
-- The CXX compiler identification is Clang 14.0.6
-- Cray Programming Environment 2.7.17 C
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /opt/cray/pe/craype/2.7.17/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Cray Programming Environment 2.7.17 CXX
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /opt/cray/pe/craype/2.7.17/bin/CC - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found CCache: /sw/crusher/spack-envs/base/opt/linux-sles15-x86_64/gcc-7.5.0/ccache-4.5.1-gq5xkxxgeqkxopbjy6gacidd36xglhqc/bin/ccache
-- Downloading ABLASTR ...
-- ABLASTR repository: https://github.com/ECP-WarpX/WarpX.git (23.02)
-- Found CCache: /sw/crusher/spack-envs/base/opt/linux-sles15-x86_64/gcc-7.5.0/ccache-4.5.1-gq5xkxxgeqkxopbjy6gacidd36xglhqc/bin/ccache
-- Downloading AMReX ...
-- AMReX repository: https://github.com/AMReX-Codes/amrex.git (23.02)
-- CMake version: 3.23.2
-- AMReX installation directory: /usr/local
-- Build type set by user to 'RelWithDebInfo'.
-- Building AMReX with AMReX_SPACEDIM = 3
-- Configuring AMReX with the following options enabled: 
--    AMReX_PRECISION = DOUBLE
--    AMReX_GPU_BACKEND = HIP
--    AMReX_MPI
--    AMReX_MPI_THREAD_MULTIPLE
--    AMReX_LINEAR_SOLVERS
--    AMReX_PARTICLES
--    AMReX_PARTICLES_PRECISION = DOUBLE
--    AMReX_TINY_PROFILE
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE  
-- Found MPI_C: /opt/cray/pe/craype/2.7.17/bin/cc (found version "3.1") 
-- Found MPI_CXX: /opt/cray/pe/craype/2.7.17/bin/CC (found version "3.1") 
-- Found MPI: TRUE (found version "3.1") found components: C CXX 
-- hip::amdhip64 is SHARED_LIBRARY
-- Performing Test HIP_CLANG_SUPPORTS_PARALLEL_JOBS
-- Performing Test HIP_CLANG_SUPPORTS_PARALLEL_JOBS - Failed
-- Found HIP: 
-- HIP: Runtime=rocclr Compiler=clang Path=/opt/rocm-5.2.0/hip
-- hip::amdhip64 is SHARED_LIBRARY
-- hip::amdhip64 is SHARED_LIBRARY
-- hip::amdhip64 is SHARED_LIBRARY
-- Performing Test LINKER_HAS_WHOLE_ARCHIVE_OFFLOAD
-- Performing Test LINKER_HAS_WHOLE_ARCHIVE_OFFLOAD - Failed
-- AMReX configuration summary: 
--    Build type               = RelWithDebInfo
--    Install directory        = /usr/local
--    C++ compiler             = /opt/cray/pe/craype/2.7.17/bin/CC
--    C++ defines              = -D__HIP_PLATFORM_HCC__=1 -D__HIP_PLATFORM_AMD__=1
--    C++ flags                = -O2 -g -DNDEBUG -I/opt/rocm-5.2.0/include -mllvm -amdgpu-early-inline-all=true -amdgpu-function-calls=false -m64 -munsafe-fp-atomics -x hip --offload-arch=gfx90a
--    C++ include paths        = -I/ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-src/Src/Base -I/ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-src/Src/Base/Parser -I/ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-src/Src/Boundary -I/ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-src/Src/AmrCore -I/ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-src/Src/LinearSolvers/MLMG -I/ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-src/Src/LinearSolvers/OpenBC -I/ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-src/Src/Particle -I/opt/rocm-5.2.0/hip/include -I/opt/rocm-5.2.0/llvm/lib/clang/14.0.0/include/.. -I/opt/rocm-5.2.0/include -I/opt/rocm-5.2.0/include/hiprand -I/opt/rocm-5.2.0/include/rocrand -I/opt/rocm-5.2.0/include/
--    Link line                = /opt/rocm-5.2.0/llvm/lib/clang/14.0.0/lib/linux/libclang_rt.builtins-x86_64.a
-- AMReX: Using version '23.02' (23.02)
-- Found Git: /usr/bin/git (found version "2.35.3") 

WarpX build configuration:
  Version: 23.02 (23.02)
  C++ Compiler: Clang 14.0.6 CrayPrgEnv
    /opt/cray/pe/craype/2.7.17/bin/CC

  Installation prefix: /usr/local
        bin: bin
        lib: lib64
    include: include
      cmake: lib64/cmake/WarpX

  Build type: RelWithDebInfo
  Build options:
    APP: OFF
    ASCENT: OFF
    COMPUTE: HIP
    DIMS: 3
    Embedded Boundary: OFF
    GPU clock timers: ON
    IPO/LTO: OFF
    LIB: OFF
    MPI: ON
    PSATD: OFF
    PRECISION: DOUBLE
    PARTICLE PRECISION: DOUBLE
    OPENPMD: OFF
    QED: OFF
    QED table generation: OFF
    SENSEI: OFF

-- ABLASTR: Using version '' (23.02)

ImpactX build configuration:
  Version: 23.02 (23.02-1-g1a134a03ef79)
  C++ Compiler: Clang 14.0.6 CrayPrgEnv
    /opt/cray/pe/craype/2.7.17/bin/CC

  Installation prefix: /usr/local
        bin: bin
        lib: lib64
    include: include
      cmake: lib64/cmake/ImpactX

  Build type: RelWithDebInfo
  Build options:
    APP: ON
    COMPUTE: HIP
    IPO/LTO: OFF
    LIB: OFF
    MPI: ON
    PRECISION: DOUBLE
    PYTHON: OFF
    OPENPMD: OFF

-- Configuring done
-- Generating done
-- Build files have been written to: /ccs/home/mgarten/src/impactx/build
[[email protected] impactx]$ cmake --build build -j 16
[  0%] Building CXX object CMakeFiles/buildInfoapp.dir/app/AMReX_buildInfo.cpp.o
[  0%] Building CXX object _deps/fetchedablastr-build/CMakeFiles/buildInfoablastr.dir/ablastr/AMReX_buildInfo.cpp.o
[  1%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_BlockMutex.cpp.o
[  1%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX.cpp.o
[  1%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_error_fi.cpp.o
[  3%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_Version.cpp.o
[  5%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_ParmParse.cpp.o
[  5%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_parmparse_fi.cpp.o
[  5%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_Utility.cpp.o
[  5%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_FileSystem.cpp.o
[  7%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_DistributionMapping.cpp.o
[  7%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_Random.cpp.o
[  7%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_ParallelDescriptor.cpp.o
[  9%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_ParallelContext.cpp.o
[  9%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_ForkJoin.cpp.o
[  9%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_VisMF.cpp.o
[ 11%] Linking CXX static library lib/libbuildInfoapp.a
[ 11%] Built target buildInfoapp
[ 13%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_AsyncOut.cpp.o
[ 15%] Linking CXX static library ../../lib/libbuildInfoablastr.a
[ 15%] Built target buildInfoablastr
[ 15%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_BackgroundThread.cpp.o
[ 17%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_Arena.cpp.o
[ 17%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_BArena.cpp.o
[ 17%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_CArena.cpp.o
[ 19%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_PArena.cpp.o
[ 19%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_BLBackTrace.cpp.o
[ 19%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_NFiles.cpp.o
[ 21%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_parstream.cpp.o
[ 21%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_FabConv.cpp.o
[ 21%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_FPC.cpp.o
[ 23%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_VectorIO.cpp.o
[ 23%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_IntConv.cpp.o
[ 25%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_Box.cpp.o
[ 25%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_BoxIterator.cpp.o
[ 25%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_IntVect.cpp.o
[ 26%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_IndexType.cpp.o
[ 26%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_Orientation.cpp.o
[ 26%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_Periodicity.cpp.o
[ 28%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_RealBox.cpp.o
[ 28%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_RealVect.cpp.o
[ 28%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_BoxList.cpp.o
[ 30%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_BoxArray.cpp.o
[ 30%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_BoxDomain.cpp.o
[ 32%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_FArrayBox.cpp.o
[ 32%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_IArrayBox.cpp.o
[ 32%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_BaseFab.cpp.o
[ 34%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_MultiFab.cpp.o
[ 34%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_MFCopyDescriptor.cpp.o
[ 34%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_iMultiFab.cpp.o
[ 36%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_FabArrayBase.cpp.o
[ 36%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_MFIter.cpp.o
[ 36%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_CoordSys.cpp.o
[ 38%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_Geometry.cpp.o
[ 38%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_MultiFabUtil.cpp.o
[ 40%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_BCRec.cpp.o
[ 40%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_PhysBCFunct.cpp.o
[ 40%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_BCUtil.cpp.o
[ 42%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_FilCC_C.cpp.o
[ 42%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_FilFC_C.cpp.o
[ 42%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_FilND_C.cpp.o
[ 44%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_NonLocalBC.cpp.o
[ 44%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_PlotFileUtil.cpp.o
[ 44%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_PlotFileDataImpl.cpp.o
[ 46%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_GpuControl.cpp.o
[ 46%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_GpuDevice.cpp.o
[ 48%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_GpuUtility.cpp.o
[ 48%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_GpuAsyncArray.cpp.o
[ 48%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_GpuElixir.cpp.o
[ 50%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_Machine.cpp.o
[ 50%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_MemPool.cpp.o
[ 50%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/Parser/AMReX_Parser.cpp.o
[ 51%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/Parser/AMReX_Parser_Exe.cpp.o
[ 51%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/Parser/AMReX_Parser_Y.cpp.o
[ 51%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/Parser/amrex_parser.lex.cpp.o
[ 53%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/Parser/amrex_parser.tab.cpp.o
[ 53%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/Parser/AMReX_IParser.cpp.o
[ 55%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/Parser/AMReX_IParser_Exe.cpp.o
[ 55%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/Parser/AMReX_IParser_Y.cpp.o
[ 55%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/Parser/amrex_iparser.lex.cpp.o
[ 57%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/Parser/amrex_iparser.tab.cpp.o
[ 57%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_BLProfiler.cpp.o
[ 57%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_TinyProfiler.cpp.o
[ 59%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Base/AMReX_MPMD.cpp.o
[ 59%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Boundary/AMReX_Mask.cpp.o
[ 59%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Boundary/AMReX_MultiMask.cpp.o
[ 61%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Boundary/AMReX_LO_BCTYPES.cpp.o
[ 61%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/AmrCore/AMReX_AmrCore.cpp.o
[ 63%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/AmrCore/AMReX_Cluster.cpp.o
[ 63%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/AmrCore/AMReX_ErrorList.cpp.o
[ 63%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/AmrCore/AMReX_FillPatchUtil.cpp.o
[ 65%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/AmrCore/AMReX_FluxRegister.cpp.o
[ 65%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/AmrCore/AMReX_InterpBase.cpp.o
[ 65%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/AmrCore/AMReX_MFInterpolater.cpp.o
[ 67%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/AmrCore/AMReX_Interpolater.cpp.o
[ 67%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/AmrCore/AMReX_TagBox.cpp.o
[ 67%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/AmrCore/AMReX_AmrMesh.cpp.o
[ 69%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/AmrCore/AMReX_InterpFaceRegister.cpp.o
[ 69%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/LinearSolvers/MLMG/AMReX_MLMG.cpp.o
[ 71%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/LinearSolvers/MLMG/AMReX_MLNodeLinOp.cpp.o
[ 71%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp.o
[ 71%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_sync.cpp.o
[ 73%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_sten.cpp.o
[ 73%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_misc.cpp.o
[ 73%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/LinearSolvers/MLMG/AMReX_MLNodeTensorLaplacian.cpp.o
[ 75%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/LinearSolvers/MLMG/AMReX_MLTensorOp.cpp.o
[ 75%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/LinearSolvers/MLMG/AMReX_MLTensorOp_grad.cpp.o
[ 75%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/LinearSolvers/MLMG/AMReX_MLEBNodeFDLaplacian.cpp.o
[ 76%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/LinearSolvers/OpenBC/AMReX_OpenBC.cpp.o
[ 76%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Particle/AMReX_TracerParticles.cpp.o
[ 78%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Particle/AMReX_ParticleMPIUtil.cpp.o
[ 78%] Building CXX object _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/Particle/AMReX_ParticleUtil.cpp.o
clang-14: /home/jenkins/llvm/lib/Target/AMDGPU/SIOptimizeVGPRLiveRange.cpp:497: void {anonymous}::SIOptimizeVGPRLiveRange::optimizeLiveRange(llvm::Register, llvm::MachineBasicBlock*, llvm::MachineBasicBlock*, llvm::MachineBasicBlock*, llvm::SmallSetVector<llvm::MachineBasicBlock*, 16>&) const: Assertion `UseMI->isPHI() && "Uses should be PHI in Endif block"' failed.
PLEASE submit a bug report to Cray and include the crash backtrace, preprocessed source, and associated run script.
Stack dump:
0.	Program arguments: /opt/cray/pe/cce/14.0.2/cce-clang/x86_64/bin/clang-14 -cc1 -triple amdgcn-amd-amdhsa -mllvm -cray-omp-opt-fork-call -mllvm -cray-omp-parallel-opt-call -mllvm -cray-openmp-rename-outlined -fcray-gpu -flocal-restrict -mllvm -cray-enhanced-asm=1 -fenhanced-asm=1 -mllvm -cray-enhanced-ir=1 -fenhanced-ir=1 -fomp-local-offload-table -ffortran -aux-triple x86_64-unknown-linux-gnu -emit-obj --mrelax-relocations -disable-free -clear-ast-before-backend -main-file-name AMReX_MLNodeLaplacian_sync.cpp -mrelocation-model pic -pic-level 1 -fhalf-no-semantic-interposition -mframe-pointer=none -fno-rounding-math -mconstructor-aliases -aux-target-cpu x86-64 -fcuda-is-device -mllvm -amdgpu-internalize-symbols -fcuda-allow-variadic-functions -fvisibility hidden -fapply-global-visibility-to-externs -mlink-builtin-bitcode /opt/rocm-5.2.0/amdgcn/bitcode/hip.bc -mlink-builtin-bitcode /opt/rocm-5.2.0/amdgcn/bitcode/ocml.bc -mlink-builtin-bitcode /opt/rocm-5.2.0/amdgcn/bitcode/ockl.bc -mlink-builtin-bitcode /opt/rocm-5.2.0/amdgcn/bitcode/oclc_daz_opt_off.bc -mlink-builtin-bitcode /opt/rocm-5.2.0/amdgcn/bitcode/oclc_unsafe_math_off.bc -mlink-builtin-bitcode /opt/rocm-5.2.0/amdgcn/bitcode/oclc_finite_only_off.bc -mlink-builtin-bitcode /opt/rocm-5.2.0/amdgcn/bitcode/oclc_correctly_rounded_sqrt_on.bc -mlink-builtin-bitcode /opt/rocm-5.2.0/amdgcn/bitcode/oclc_wavefrontsize64_on.bc -mlink-builtin-bitcode /opt/rocm-5.2.0/amdgcn/bitcode/oclc_isa_version_90a.bc -mlink-builtin-bitcode /opt/rocm-5.2.0/amdgcn/bitcode/oclc_abi_version_400.bc -target-cpu gfx90a -mllvm -treat-scalable-fixed-error-as-warning -debug-info-kind=constructor -dwarf-version=5 -debugger-tuning=gdb -resource-dir /opt/cray/pe/cce/14.0.2/cce-clang/x86_64/lib/clang/14.0.6 -dependency-file CMakeFiles/amrex.dir/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_sync.cpp.o.d -MT _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_sync.cpp.o -sys-header-deps -internal-isystem /opt/cray/pe/cce/14.0.2/cce-clang/x86_64/lib/clang/14.0.6/include/cuda_wrappers -idirafter /opt/rocm-5.2.0/include -include __clang_hip_runtime_wrapper.h -isystem /opt/rocm-5.2.0/hip/include -isystem /opt/rocm-5.2.0/llvm/lib/clang/14.0.0/include/.. -isystem /opt/rocm-5.2.0/include/hiprand -isystem /opt/rocm-5.2.0/include/rocrand -isystem /opt/cray/pe/cce/14.0.2/cce-clang/x86_64/lib/clang/14.0.6/include -isystem /opt/cray/pe/cce/14.0.2/cce/x86_64/include/craylibs -D __CRAY_X86_TRENTO -D __CRAY_AMD_GFX90A -D __CRAYXT_COMPUTE_LINUX_TARGET -D __HIP_PLATFORM_AMD__=1 -D __HIP_PLATFORM_HCC__=1 -I /ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-src/Src/Base -I /ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-src/Src/Base/Parser -I /ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-src/Src/Boundary -I /ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-src/Src/AmrCore -I /ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-src/Src/LinearSolvers/MLMG -I /ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-src/Src/LinearSolvers/OpenBC -I /ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-src/Src/Particle -I /ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-build -I /opt/rocm-5.2.0/include -D NDEBUG -I /opt/cray/pe/mpich/8.1.18/ofi/cray/10.0/include -I /opt/cray/pe/libsci/22.08.1.1/CRAY/9.0/x86_64/include -I /opt/cray/pe/hdf5-parallel/1.12.1.5/crayclang/14.0/include -I /opt/cray/pe/pmi/6.1.3/include -I /opt/cray/pe/dsmml/0.2.2/dsmml//include -I /opt/cray/xpmem/2.4.4-2.3_11.2__gff0e1d9.shasta/include -internal-isystem /opt/cray/pe/gcc/10.3.0/snos/lib/gcc/x86_64-suse-linux/10.3.0/../../../../include/g++ -internal-isystem /opt/cray/pe/gcc/10.3.0/snos/lib/gcc/x86_64-suse-linux/10.3.0/../../../../include/g++/x86_64-suse-linux -internal-isystem /opt/cray/pe/gcc/10.3.0/snos/lib/gcc/x86_64-suse-linux/10.3.0/../../../../include/g++/backward -internal-isystem /opt/cray/pe/gcc/10.3.0/snos/lib/gcc/x86_64-suse-linux/10.3.0/../../../../include/g++ -internal-isystem /opt/cray/pe/gcc/10.3.0/snos/lib/gcc/x86_64-suse-linux/10.3.0/../../../../include/g++/x86_64-suse-linux -internal-isystem /opt/cray/pe/gcc/10.3.0/snos/lib/gcc/x86_64-suse-linux/10.3.0/../../../../include/g++/backward -internal-isystem /opt/cray/pe/cce/14.0.2/cce-clang/x86_64/lib/clang/14.0.6/include -internal-isystem /usr/local/include -internal-isystem /opt/cray/pe/gcc/10.3.0/snos/lib/gcc/x86_64-suse-linux/10.3.0/../../../../x86_64-suse-linux/include -internal-externc-isystem /include -internal-externc-isystem /usr/include -internal-isystem /opt/cray/pe/cce/14.0.2/cce-clang/x86_64/lib/clang/14.0.6/include -internal-isystem /usr/local/include -internal-isystem /opt/cray/pe/gcc/10.3.0/snos/lib/gcc/x86_64-suse-linux/10.3.0/../../../../x86_64-suse-linux/include -internal-externc-isystem /include -internal-externc-isystem /usr/include -O2 -std=c++17 -fdeprecated-macro -fno-autolink -fdebug-compilation-dir=/ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-build/Src -ferror-limit 19 -fhip-new-launch-api -fgnuc-version=4.2.1 -fcxx-exceptions -fexceptions -mllvm -amdgpu-early-inline-all=true -mllvm -amdgpu-function-calls=false -cuid=a1db559448e604e2 -fcuda-allow-variadic-functions -munsafe-fp-atomics -faddrsig -D__GCC_HAVE_DWARF2_CFI_ASM=1 -o /tmp/AMReX_MLNodeLaplacian_sync-f6a335/AMReX_MLNodeLaplacian_sync-gfx90a.o -x hip /ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-src/Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_sync.cpp
1.	<eof> parser at end of file
2.	Code generation
3.	Running pass 'CallGraph Pass Manager' on module '/ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-src/Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_sync.cpp'.
4.	Running pass 'SI Optimize VGPR LiveRange' on function '@_ZN5amrex13launch_globalILi256EZNS_11ParallelForILi256EZNKS_15MLNodeLaplacian6refluxEiRNS_8MultiFabERKS3_S6_S4_S4_S6_EUliiiE3_EENSt9enable_ifIXsr19MaybeDeviceRunnableIT0_EE5valueEvE4typeERKNS_3Gpu10KernelInfoERKNS_3BoxEOS9_EUlvE_EEvS9_'
 #0 0x00000000038258cf PrintStackTraceSignalHandler(void*) Signals.cpp:0:0
 #1 0x00000000038229ed SignalHandler(int) Signals.cpp:0:0
 #2 0x00007fbdc691a8bf (/lib64/libpthread.so.0+0x168bf)
 #3 0x00007fbdc55fecda raise (/lib64/libc.so.6+0x4acda)
 #4 0x00007fbdc5600374 abort (/lib64/libc.so.6+0x4c374)
 #5 0x00007fbdc55f6cd9 __assert_fail_base (/lib64/libc.so.6+0x42cd9)
 #6 0x00007fbdc55f6d61 __assert_fail (/lib64/libc.so.6+0x42d61)
 #7 0x000000000126ee3e (anonymous namespace)::SIOptimizeVGPRLiveRange::optimizeLiveRange(llvm::Register, llvm::MachineBasicBlock*, llvm::MachineBasicBlock*, llvm::MachineBasicBlock*, llvm::SmallSetVector<llvm::MachineBasicBlock*, 16u>&) const SIOptimizeVGPRLiveRange.cpp:0:0
 #8 0x0000000001272e93 (anonymous namespace)::SIOptimizeVGPRLiveRange::runOnMachineFunction(llvm::MachineFunction&) SIOptimizeVGPRLiveRange.cpp:0:0
 #9 0x00000000028bdacf llvm::MachineFunctionPass::runOnFunction(llvm::Function&) (/opt/cray/pe/cce/14.0.2/cce-clang/x86_64/bin/clang-14+0x28bdacf)
#10 0x0000000002dea470 llvm::FPPassManager::runOnFunction(llvm::Function&) (/opt/cray/pe/cce/14.0.2/cce-clang/x86_64/bin/clang-14+0x2dea470)
#11 0x00000000024c7319 (anonymous namespace)::CGPassManager::runOnModule(llvm::Module&) CallGraphSCCPass.cpp:0:0
#12 0x0000000002dec0c6 llvm::legacy::PassManagerImpl::run(llvm::Module&) (/opt/cray/pe/cce/14.0.2/cce-clang/x86_64/bin/clang-14+0x2dec0c6)
#13 0x0000000003badbea clang::EmitBackendOutput(clang::DiagnosticsEngine&, clang::HeaderSearchOptions const&, clang::CodeGenOptions const&, clang::TargetOptions const&, clang::LangOptions const&, clang::FrontendOptions const&, llvm::StringRef, llvm::Module*, clang::BackendAction, std::unique_ptr<llvm::raw_pwrite_stream, std::default_delete<llvm::raw_pwrite_stream> >) (/opt/cray/pe/cce/14.0.2/cce-clang/x86_64/bin/clang-14+0x3badbea)
#14 0x0000000004a84d55 clang::BackendConsumer::HandleTranslationUnit(clang::ASTContext&) (/opt/cray/pe/cce/14.0.2/cce-clang/x86_64/bin/clang-14+0x4a84d55)
#15 0x0000000005601088 clang::ParseAST(clang::Sema&, bool, bool) (/opt/cray/pe/cce/14.0.2/cce-clang/x86_64/bin/clang-14+0x5601088)
#16 0x000000000436c7b8 clang::FrontendAction::Execute() (/opt/cray/pe/cce/14.0.2/cce-clang/x86_64/bin/clang-14+0x436c7b8)
#17 0x00000000042ec625 clang::CompilerInstance::ExecuteAction(clang::FrontendAction&) (/opt/cray/pe/cce/14.0.2/cce-clang/x86_64/bin/clang-14+0x42ec625)
#18 0x0000000004433a8a clang::ExecuteCompilerInvocation(clang::CompilerInstance*) (/opt/cray/pe/cce/14.0.2/cce-clang/x86_64/bin/clang-14+0x4433a8a)
#19 0x0000000000df2365 cc1_main(llvm::ArrayRef<char const*>, char const*, void*) (/opt/cray/pe/cce/14.0.2/cce-clang/x86_64/bin/clang-14+0xdf2365)
#20 0x0000000000defa6b ExecuteCC1Tool(llvm::SmallVectorImpl<char const*>&) driver.cpp:0:0
#21 0x0000000000d2eb86 main (/opt/cray/pe/cce/14.0.2/cce-clang/x86_64/bin/clang-14+0xd2eb86)
#22 0x00007fbdc55e92bc __libc_start_main (/lib64/libc.so.6+0x352bc)
#23 0x0000000000ded0f9 _start /home/abuild/rpmbuild/BUILD/glibc-2.26/csu/../sysdeps/x86_64/start.S:120:0
clang-14: error: unable to execute command: Aborted (core dumped)
clang-14: error: clang frontend command failed due to signal (use -v to see invocation)
Cray clang version 14.0.2  (ecfd9ef4dfd5696cd449133c0da0293d503c2f21)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/cray/pe/cce/14.0.2/cce-clang/x86_64/share/../bin
clang-14: note: diagnostic msg: Error generating preprocessed source(s).
gmake[2]: *** [_deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/build.make:1406: _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/LinearSolvers/MLMG/AMReX_MLNodeLaplacian_sync.cpp.o] Error 254
gmake[2]: *** Waiting for unfinished jobs....
clang-14: /home/jenkins/llvm/lib/Target/AMDGPU/SIOptimizeVGPRLiveRange.cpp:497: void {anonymous}::SIOptimizeVGPRLiveRange::optimizeLiveRange(llvm::Register, llvm::MachineBasicBlock*, llvm::MachineBasicBlock*, llvm::MachineBasicBlock*, llvm::SmallSetVector<llvm::MachineBasicBlock*, 16>&) const: Assertion `UseMI->isPHI() && "Uses should be PHI in Endif block"' failed.
PLEASE submit a bug report to Cray and include the crash backtrace, preprocessed source, and associated run script.
Stack dump:
0.	Program arguments: /opt/cray/pe/cce/14.0.2/cce-clang/x86_64/bin/clang-14 -cc1 -triple amdgcn-amd-amdhsa -mllvm -cray-omp-opt-fork-call -mllvm -cray-omp-parallel-opt-call -mllvm -cray-openmp-rename-outlined -fcray-gpu -flocal-restrict -mllvm -cray-enhanced-asm=1 -fenhanced-asm=1 -mllvm -cray-enhanced-ir=1 -fenhanced-ir=1 -fomp-local-offload-table -ffortran -aux-triple x86_64-unknown-linux-gnu -emit-obj --mrelax-relocations -disable-free -clear-ast-before-backend -main-file-name AMReX_MLNodeLaplacian.cpp -mrelocation-model pic -pic-level 1 -fhalf-no-semantic-interposition -mframe-pointer=none -fno-rounding-math -mconstructor-aliases -aux-target-cpu x86-64 -fcuda-is-device -mllvm -amdgpu-internalize-symbols -fcuda-allow-variadic-functions -fvisibility hidden -fapply-global-visibility-to-externs -mlink-builtin-bitcode /opt/rocm-5.2.0/amdgcn/bitcode/hip.bc -mlink-builtin-bitcode /opt/rocm-5.2.0/amdgcn/bitcode/ocml.bc -mlink-builtin-bitcode /opt/rocm-5.2.0/amdgcn/bitcode/ockl.bc -mlink-builtin-bitcode /opt/rocm-5.2.0/amdgcn/bitcode/oclc_daz_opt_off.bc -mlink-builtin-bitcode /opt/rocm-5.2.0/amdgcn/bitcode/oclc_unsafe_math_off.bc -mlink-builtin-bitcode /opt/rocm-5.2.0/amdgcn/bitcode/oclc_finite_only_off.bc -mlink-builtin-bitcode /opt/rocm-5.2.0/amdgcn/bitcode/oclc_correctly_rounded_sqrt_on.bc -mlink-builtin-bitcode /opt/rocm-5.2.0/amdgcn/bitcode/oclc_wavefrontsize64_on.bc -mlink-builtin-bitcode /opt/rocm-5.2.0/amdgcn/bitcode/oclc_isa_version_90a.bc -mlink-builtin-bitcode /opt/rocm-5.2.0/amdgcn/bitcode/oclc_abi_version_400.bc -target-cpu gfx90a -mllvm -treat-scalable-fixed-error-as-warning -debug-info-kind=constructor -dwarf-version=5 -debugger-tuning=gdb -resource-dir /opt/cray/pe/cce/14.0.2/cce-clang/x86_64/lib/clang/14.0.6 -dependency-file CMakeFiles/amrex.dir/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp.o.d -MT _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp.o -sys-header-deps -internal-isystem /opt/cray/pe/cce/14.0.2/cce-clang/x86_64/lib/clang/14.0.6/include/cuda_wrappers -idirafter /opt/rocm-5.2.0/include -include __clang_hip_runtime_wrapper.h -isystem /opt/rocm-5.2.0/hip/include -isystem /opt/rocm-5.2.0/llvm/lib/clang/14.0.0/include/.. -isystem /opt/rocm-5.2.0/include/hiprand -isystem /opt/rocm-5.2.0/include/rocrand -isystem /opt/cray/pe/cce/14.0.2/cce-clang/x86_64/lib/clang/14.0.6/include -isystem /opt/cray/pe/cce/14.0.2/cce/x86_64/include/craylibs -D __CRAY_X86_TRENTO -D __CRAY_AMD_GFX90A -D __CRAYXT_COMPUTE_LINUX_TARGET -D __HIP_PLATFORM_AMD__=1 -D __HIP_PLATFORM_HCC__=1 -I /ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-src/Src/Base -I /ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-src/Src/Base/Parser -I /ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-src/Src/Boundary -I /ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-src/Src/AmrCore -I /ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-src/Src/LinearSolvers/MLMG -I /ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-src/Src/LinearSolvers/OpenBC -I /ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-src/Src/Particle -I /ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-build -I /opt/rocm-5.2.0/include -D NDEBUG -I /opt/cray/pe/mpich/8.1.18/ofi/cray/10.0/include -I /opt/cray/pe/libsci/22.08.1.1/CRAY/9.0/x86_64/include -I /opt/cray/pe/hdf5-parallel/1.12.1.5/crayclang/14.0/include -I /opt/cray/pe/pmi/6.1.3/include -I /opt/cray/pe/dsmml/0.2.2/dsmml//include -I /opt/cray/xpmem/2.4.4-2.3_11.2__gff0e1d9.shasta/include -internal-isystem /opt/cray/pe/gcc/10.3.0/snos/lib/gcc/x86_64-suse-linux/10.3.0/../../../../include/g++ -internal-isystem /opt/cray/pe/gcc/10.3.0/snos/lib/gcc/x86_64-suse-linux/10.3.0/../../../../include/g++/x86_64-suse-linux -internal-isystem /opt/cray/pe/gcc/10.3.0/snos/lib/gcc/x86_64-suse-linux/10.3.0/../../../../include/g++/backward -internal-isystem /opt/cray/pe/gcc/10.3.0/snos/lib/gcc/x86_64-suse-linux/10.3.0/../../../../include/g++ -internal-isystem /opt/cray/pe/gcc/10.3.0/snos/lib/gcc/x86_64-suse-linux/10.3.0/../../../../include/g++/x86_64-suse-linux -internal-isystem /opt/cray/pe/gcc/10.3.0/snos/lib/gcc/x86_64-suse-linux/10.3.0/../../../../include/g++/backward -internal-isystem /opt/cray/pe/cce/14.0.2/cce-clang/x86_64/lib/clang/14.0.6/include -internal-isystem /usr/local/include -internal-isystem /opt/cray/pe/gcc/10.3.0/snos/lib/gcc/x86_64-suse-linux/10.3.0/../../../../x86_64-suse-linux/include -internal-externc-isystem /include -internal-externc-isystem /usr/include -internal-isystem /opt/cray/pe/cce/14.0.2/cce-clang/x86_64/lib/clang/14.0.6/include -internal-isystem /usr/local/include -internal-isystem /opt/cray/pe/gcc/10.3.0/snos/lib/gcc/x86_64-suse-linux/10.3.0/../../../../x86_64-suse-linux/include -internal-externc-isystem /include -internal-externc-isystem /usr/include -O2 -std=c++17 -fdeprecated-macro -fno-autolink -fdebug-compilation-dir=/ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-build/Src -ferror-limit 19 -fhip-new-launch-api -fgnuc-version=4.2.1 -fcxx-exceptions -fexceptions -mllvm -amdgpu-early-inline-all=true -mllvm -amdgpu-function-calls=false -cuid=2f96d5e7b873b0d -fcuda-allow-variadic-functions -munsafe-fp-atomics -faddrsig -D__GCC_HAVE_DWARF2_CFI_ASM=1 -o /tmp/AMReX_MLNodeLaplacian-3587ac/AMReX_MLNodeLaplacian-gfx90a.o -x hip /ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-src/Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp
1.	<eof> parser at end of file
2.	Code generation
3.	Running pass 'CallGraph Pass Manager' on module '/ccs/home/mgarten/src/impactx/build/_deps/fetchedamrex-src/Src/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp'.
4.	Running pass 'SI Optimize VGPR LiveRange' on function '@_ZN5amrex13launch_globalILi256EZNS_11ParallelForILi256EZNKS_15MLNodeLaplacian21restrictInteriorNodesEiRNS_8MultiFabES4_EUliiiE3_EENSt9enable_ifIXsr19MaybeDeviceRunnableIT0_EE5valueEvE4typeERKNS_3Gpu10KernelInfoERKNS_3BoxEOS7_EUlvE_EEvS7_'
 #0 0x00000000038258cf PrintStackTraceSignalHandler(void*) Signals.cpp:0:0
 #1 0x00000000038229ed SignalHandler(int) Signals.cpp:0:0
 #2 0x00007fba2efcd8bf (/lib64/libpthread.so.0+0x168bf)
 #3 0x00007fba2dcb1cda raise (/lib64/libc.so.6+0x4acda)
 #4 0x00007fba2dcb3374 abort (/lib64/libc.so.6+0x4c374)
 #5 0x00007fba2dca9cd9 __assert_fail_base (/lib64/libc.so.6+0x42cd9)
 #6 0x00007fba2dca9d61 __assert_fail (/lib64/libc.so.6+0x42d61)
 #7 0x000000000126ee3e (anonymous namespace)::SIOptimizeVGPRLiveRange::optimizeLiveRange(llvm::Register, llvm::MachineBasicBlock*, llvm::MachineBasicBlock*, llvm::MachineBasicBlock*, llvm::SmallSetVector<llvm::MachineBasicBlock*, 16u>&) const SIOptimizeVGPRLiveRange.cpp:0:0
 #8 0x0000000001272e93 (anonymous namespace)::SIOptimizeVGPRLiveRange::runOnMachineFunction(llvm::MachineFunction&) SIOptimizeVGPRLiveRange.cpp:0:0
 #9 0x00000000028bdacf llvm::MachineFunctionPass::runOnFunction(llvm::Function&) (/opt/cray/pe/cce/14.0.2/cce-clang/x86_64/bin/clang-14+0x28bdacf)
#10 0x0000000002dea470 llvm::FPPassManager::runOnFunction(llvm::Function&) (/opt/cray/pe/cce/14.0.2/cce-clang/x86_64/bin/clang-14+0x2dea470)
#11 0x00000000024c7319 (anonymous namespace)::CGPassManager::runOnModule(llvm::Module&) CallGraphSCCPass.cpp:0:0
#12 0x0000000002dec0c6 llvm::legacy::PassManagerImpl::run(llvm::Module&) (/opt/cray/pe/cce/14.0.2/cce-clang/x86_64/bin/clang-14+0x2dec0c6)
#13 0x0000000003badbea clang::EmitBackendOutput(clang::DiagnosticsEngine&, clang::HeaderSearchOptions const&, clang::CodeGenOptions const&, clang::TargetOptions const&, clang::LangOptions const&, clang::FrontendOptions const&, llvm::StringRef, llvm::Module*, clang::BackendAction, std::unique_ptr<llvm::raw_pwrite_stream, std::default_delete<llvm::raw_pwrite_stream> >) (/opt/cray/pe/cce/14.0.2/cce-clang/x86_64/bin/clang-14+0x3badbea)
#14 0x0000000004a84d55 clang::BackendConsumer::HandleTranslationUnit(clang::ASTContext&) (/opt/cray/pe/cce/14.0.2/cce-clang/x86_64/bin/clang-14+0x4a84d55)
#15 0x0000000005601088 clang::ParseAST(clang::Sema&, bool, bool) (/opt/cray/pe/cce/14.0.2/cce-clang/x86_64/bin/clang-14+0x5601088)
#16 0x000000000436c7b8 clang::FrontendAction::Execute() (/opt/cray/pe/cce/14.0.2/cce-clang/x86_64/bin/clang-14+0x436c7b8)
#17 0x00000000042ec625 clang::CompilerInstance::ExecuteAction(clang::FrontendAction&) (/opt/cray/pe/cce/14.0.2/cce-clang/x86_64/bin/clang-14+0x42ec625)
#18 0x0000000004433a8a clang::ExecuteCompilerInvocation(clang::CompilerInstance*) (/opt/cray/pe/cce/14.0.2/cce-clang/x86_64/bin/clang-14+0x4433a8a)
#19 0x0000000000df2365 cc1_main(llvm::ArrayRef<char const*>, char const*, void*) (/opt/cray/pe/cce/14.0.2/cce-clang/x86_64/bin/clang-14+0xdf2365)
#20 0x0000000000defa6b ExecuteCC1Tool(llvm::SmallVectorImpl<char const*>&) driver.cpp:0:0
#21 0x0000000000d2eb86 main (/opt/cray/pe/cce/14.0.2/cce-clang/x86_64/bin/clang-14+0xd2eb86)
#22 0x00007fba2dc9c2bc __libc_start_main (/lib64/libc.so.6+0x352bc)
#23 0x0000000000ded0f9 _start /home/abuild/rpmbuild/BUILD/glibc-2.26/csu/../sysdeps/x86_64/start.S:120:0
clang-14: error: unable to execute command: Aborted
clang-14: error: clang frontend command failed due to signal (use -v to see invocation)
Cray clang version 14.0.2  (ecfd9ef4dfd5696cd449133c0da0293d503c2f21)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /opt/cray/pe/cce/14.0.2/cce-clang/x86_64/share/../bin
clang-14: note: diagnostic msg: Error generating preprocessed source(s).
gmake[2]: *** [_deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/build.make:1392: _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/LinearSolvers/MLMG/AMReX_MLNodeLaplacian.cpp.o] Error 254
gmake[1]: *** [CMakeFiles/Makefile2:1978: _deps/fetchedamrex-build/Src/CMakeFiles/amrex.dir/all] Error 2
gmake: *** [Makefile:146: all] Error 2
[[email protected] impactx]$ 

Edit 2023/02/08: removed -DImpactX_PYTHON=ON in description here to make it simpler; add download instructions

Distributions: Move to ABLASTR

The particle distributions implemented in ImpactX are s-based, but can be re-interpreted also in t-based, mostly by passing different values for s/z. Thus, we can move their implementations (not their initializations) to ABLASTR to share them with WarpX.

OpenMP Support

When compiling with the OpenMP backend, we do not yet use particle tiling it seems.
With our current distribution strategy, we start with one block per device.

Variable name: pti can be confusing

We use pti for the ParIter of AMReX.

@cemitch99 mentioned that this can be confusing for new readers, because it might also be the initial value of pt in a readers mind.

We could change this throughout the code base and call these variables pit, p_it, par_it or par_iter or so.

Embedded Boundaries

Activate the option to use embedded boundaries for our space charge solver.

We already have the interfaces exposed in ABLASTR for the Poisson solver.

  • call APIs
  • ensure EB does not create overheads in memory or computation if not used at runtime
  • add example with implicit boundary function
  • add example with STL (CAD) file

Analysis Test Scripts: Check <z> and pt

We should check if our beam particles actually moved, instead of only checking 2nd moments.

We often check stationary and periodic solutions. If we only check the 2nd moments, we will not notice if particles did not move at all. For instance, in the development of #214 we accidentally did a zero-element simulation which passed all tests and only failed during plotting.

Thus, we should check:

  • the expected phase change on pt (and/or pz) of the reference particle
  • the expected average position of the particle beam.

cc @cemitch99 @n01r and me

CUDA PyTest Segfaults on Perlmutter (NERSC)

At the moment (23.01), it looks like the ctest runs of pytest for ImpactX segfault on Perlmutter when doing multiple finalize-init test cycles.

This is likely due to a leaking MultiFab object (or another object that uses the AMReX arena and is not fully cleaned when finalize of a test is called).

Space-Charge: MR

Based on #101, ensure that MR capabilities work.

  • decide on particle handling, e.g., only live on finest level (yes)
  • expose MR control in the user interface for at least one level
  • implement synchronization of charge from fine levels to coarse levels

Implementation of soft-edge fringe fields

Use symplectic integrators for the treatment of soft-edge quads, solenoids, (possibly) dipoles, and any other elements that may require this treatment for SciDAC applications.

Add Data-Driven Element

Add a first data-driven element (map).

  • training: read-only element
  • inference: manipulate particles

Design for collective effects: take into account to also pass global beam parameters, such as beam charge/current and beam moments.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.