Hello everyone,

I'm new to using PeleLMeX and I'm trying to simulate the flamesheet model using multiple processors. However, I've noticed that some of the files generated during the simulation have names like 'plt00000.old.0782200/', which is causing issues when I try to visualize the results using visIt.

Does anyone have any advice or best practices for avoiding these kinds of issues when working with multiple processors? Any guidance you can offer would be greatly appreciated.

Thank you!

Is it possible to read input data from hdf5 file?

In PeleLM/PeleLMex， can the walls in the figure be modeled and meshed? Or can this wall be embedded?
This wall is built based on the expression: f=cos(2pai x)*cos(2pai z), pai =3.14.

Hi,
If I run with the incompressible flag switched on, the code segfaults. I tracked it down to the line in the Utils that tries to get the reactions MF without a flag check (e.g. m_do_react or m_incompressible, referenced below). I could fix it and put in a pull request but figured someone could add this quick fix without needing a full pull request.

Thomas

Hello,
Regarding the Lewis number. I would like to impose a constant number, for example, Le = 0.8.
I tried to consult the documentation, but I couldn't find an explanation on this matter.
Is it possible to set a non-unity Lewis number?

I am facing a compilation issue within a cluster, specifically related to the creation of the executable file.

I created a new mechanism within the 'Mechanism' section of PelePhysics. This mechanism works well on my local desktop for simplified cases, allowing me to obtain the .exe file and run computations successfully.

However, when I attempt to compile it on the cluster, I am unable to generate the .exe file, and I receive the following error message: /PeleLMeX-1/Submodules/PelePhysics/ThirdParty/INSTALL/gnu/lib/libsundials_cvode.so: undefined reference to `pow@GLIBC_2.29'
collect2: error: ld returned 1 exit status
make: *** [/PeleLMeX-1/Submodules/amrex/Tools/GNUMake/Make.rules:56: PeleLMeX2d.gnu.ex] Error 1

Interestingly, when I switch to a different mechanism in the GNUmakefile, the compilation process continues until it reaches the linking stage and generates the .exe file.

Do you have any suggestions on how to avoid such issues using the new mechanism, particularly when working with a cluster?

Hi all,

I am looking to run a buoyancy-driven flow that is open to atmospheric conditions. However, when running a simulation with boundary conditions specified as "Outflow" normal to the direction of gravity, the hydrostatic condition induced by the gravitational field disappears.

To show this most clearly, starting from the HotBubble tutorial case, the following conditions

geometry.is_periodic = 1 0               # For each dir, 0: non-perio, 1: periodic
peleLM.lo_bc = Interior NoSlipWallAdiab
peleLM.hi_bc = Interior Outflow

produced an avg_pressure field (using yt)

as expected for the first snapshot with a hydrostatic condition.

Using

geometry.is_periodic = 0 0               # For each dir, 0: non-perio, 1: periodic
peleLM.lo_bc = NoSlipWallAdiab NoSlipWallAdiab
peleLM.hi_bc = NoSlipWallAdiab Outflow

produced

Again, this is expected.

And finally, using

geometry.is_periodic = 0 0               # For each dir, 0: non-perio, 1: periodic
peleLM.lo_bc = Outflow NoSlipWallAdiab
peleLM.hi_bc = Outflow Outflow

produced

This last image I would have expected to be quite similar to the other two. Is there a physical justification for this behavior or a bug? If the former, is there a way to introduce the hydrostatic condition?

Thank you very much!!

Hi all,

I am looking to use HeatRelease as a quantity to refine the grid on, and looking through the tutorials, it looks like the swirl case does exactly this. However, when I try to do this, I get an error

amrex::Error::42::PeleLM::taggingSetup(): unknown variable field for criteria HR

Is HeatRelease a supported tagging field? If not, is this something you want to include in the future?

Thanks!

if we have a cylinder in the domain, can i fix the mass fraction of species on the cylinder like fix the temperature?

Hi all,

Is there any interest in combining the static and adaptive mesh refinement approaches? I am finding many instances where I would like to adaptively refine only a specific region of the flow with a certain criteria. It would look something like:

amr.gradrho.max_level = 2
amr.gradrho.adjacent_difference_greater = 0.01
amr.gradrho.field_name = density
amr.gradrho.in_box_lo = -1.0 -1.0 0.0
amr.gradrho.in_box_hi =  1.0  1.0 2.5

This would only tag density differences within the specified in_box region.

Thanks!
Mike

Hi everyone,

I'm trying to solve the FlameSheet problem, which is a RegTest within PeleLMeX. I have a problem that when I apply the OMP option of GNUmakefile, the problem does not converge and hangs with errors. I don't have any problems when using MPI only. Is there a problem with the OMP option?

The compilation options are as follows

Compilation

COMP = gnu
USE_MPI = TRUE
USE_OMP = TRUE
USE_CUDA = FALSE
USE_HIP = FALSE

@esclapez I made a case that EB_FlamePastCylinder in PeleLMeX (because I want to use the isothermal boundary of EB), the calculation is very fast only flow, but adding the reaction, the calculation becomes very slow. This is my input file：
input.txt

Hello. I am trying to replicate the "Non-reacting flow past a cylinder" tutorial provided by amrex-combustion under PeleLMeX. I did everything as specified in the tutorial, the only change that I made was turning off the MPI, by doing USE_MPI = FALSE in GNUmakefile.
When I try to run the case by giving the command "./PeleLMeX2d.gnu.ex input.2d-Re500", it gives this error -

Doing initial projection(s)

amrex::Abort::0::MLMGBndry::setBoxBC: Unknown LinOpBCType !!!
SIGABRT
See Backtrace.0 file for details

Can you please tell me where am I going wrong?

Thank you for your time.

Hello, I am able to run a low Mach case with a fairly complex geometry with EB on CPUs. However, the same setup doesn't run on GPUs. It complains of illegal memory access. My case has isothermal walls.
After ruling out all possibilities, I have narrowed down the issue: switching the flag to peleLM.isothermal_EB = 0 runs seamlessly, but setting it to 1 results in the above mentioned error.

The Backtrace file points to PeleLMUtils.cpp (line 333 - function - intFluxDivergenceLevelEB) where this issue may have arisen first.

Is there a fix for this? Looking forward to some help on this.

Thanks!

Taaresh

How to add a derivative of the mixture fraction or progress variable over time

I would like to perform post-processing in Python without going through PeleAnalysis.

Firstly, is this possible? If yes, I would like to know if it's possible to read the files, for instance, to create an animation of the movement of my maximum Heat Release within the domain if we can achieve up to 10 levels of AMR.

LMeX repo is close to 200MB, something is probably up, where large files have been commited.

Hello,

I am reaching out to you because I have just switched to another cluster to run my PeleLMeX calculations. I have gathered all the files and made sure I have everything needed for compilation. To do this, I started by running 'make TPLrealclean' and then 'make TPL.'

After executing 'make TPL,' the intention is to create the SUNDIALS library from GitHub. The issue I am encountering with the new cluster is that GitHub is not accessible! As a result, the compilation of 'make TPL' fails, and I receive the following message:

$ make TPL
Loading /ccc/work/cont003/gen7456/bazharza/PeleLMeX/Submodules/amrex/Tools/GNUMake/comps/gnu.mak...
Loading /ccc/work/cont003/gen7456/bazharza/PeleLMeX/Submodules/amrex/Tools/GNUMake/sites/Make.unknown...
Loading /ccc/work/cont003/gen7456/bazharza/PeleLMeX/Submodules/amrex/Tools/GNUMake/packages/Make.sundials...
GNUmakefile:265: warning: overriding recipe for target '/'
GNUmakefile:240: warning: ignoring old recipe for target '/'
==> Building SUNDIALS library
make[1]: Entering directory '/ccc/work/cont003/gen7456/bazharza/PeleLMeX/Submodules/PelePhysics/ThirdParty'
Loading /ccc/work/cont003/gen7456/bazharza/PeleLMeX/Submodules/amrex/Tools/GNUMake/comps/gnu.mak...
Loading /ccc/work/cont003/gen7456/bazharza/PeleLMeX/Submodules/amrex/Tools/GNUMake/sites/Make.unknown...
GNUmakefile:265: warning: overriding recipe for target '/'
GNUmakefile:240: warning: ignoring old recipe for target '/'
cp: cannot stat '/cmb138/software/v6.5.0.tar.gz': No such file or directory
--2024-01-17 17:22:36-- https://github.com/LLNL/sundials/archive/refs/tags/v6.5.0.tar.gz
Resolving github.com (github.com)... 140.82.121.3
Connecting to github.com (github.com)|140.82.121.3|:443... failed: Connection timed out.
Retrying.

Could you please advise on how to overcome this compilation step and create the SUNDIALS library without using GitHub?

Thank you in advance.

Recursive git clone was used to perform installing. At folder Exec/RegTests/EB_FlowPastCylinder, make gives

Loading ../../../Submodules/amrex/Tools/GNUMake/comps/gnu.mak...
Loading ../../../Submodules/amrex/Tools/GNUMake/sites/Make.unknown...
Loading ../../../Submodules/amrex/Tools/GNUMake/packages/Make.sundials...
GNUmakefile:265: warning: overriding recipe for target '/'
GNUmakefile:240: warning: ignoring old recipe for target '/'
Generating AMReX_Config.H ...
Generating AMReX_Version.H ...
Compiling AMReX_NVector_MultiFab.cpp ...
In file included from ../../../Submodules/amrex/Src/Extern/SUNDIALS/AMReX_NVector_MultiFab.H:19,
                 from ../../../Submodules/amrex/Src/Extern/SUNDIALS/AMReX_NVector_MultiFab.cpp:12:
../../../Submodules/amrex/Src/Extern/SUNDIALS/AMReX_Sundials_Core.H:6:10: fatal error: sundials/sundials_context.h: No such file or directory
    6 | #include <sundials/sundials_context.h>
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
make: *** [../../../Submodules/amrex/Tools/GNUMake/Make.rules:262: tmp_build_dir/o/2d.gnu.MPI.EXE/AMReX_NVector_MultiFab.o] Error 1

I would like to perform post-processing on plt files, but I am having trouble understanding the data storage format within these files.
Specifically, I want to extract the temperature field and calculate the temperature gradient using Python. Is it possible to achieve this?

The numerical setup is shown in the provided picture.
When I set the inlet air to be 1500K, the case will run normally and finally reach a steady state with a diffusion flame.
However, when I change the inlet air to be 1150K (According to http://dx.doi.org/10.1016/j.proci.2016.06.101, it will be a auto-ignition case with Cool flame), PeleLMeX reports an error and returns the following code.

I would like to ask if PeleLMeX can be used to calculate auto-ignition cases. What are the possible meanings and solutions of this error? Thanks for any reply

Host Name: ibnode74
=== If no file names and line numbers are shown below, one can run
addr2line -Cpfie my_exefile my_line_address
to convert my_line_address (e.g., 0x4a6b) into file name and line number.
Or one can use amrex/Tools/Backtrace/parse_bt.py.

=== Please note that the line number reported by addr2line may not be accurate.
One can use
readelf -wl my_exefile | grep my_line_address'
to find out the offset for that line.

0: ./PeleLMeX2d.gnu.MPI.ex() [0xf1c1c6]
amrex::BLBackTrace::print_backtrace_info(_IO_FILE*)
/share/home/zhengjian/Pele/amrex/Src/Base/AMReX_BLBackTrace.cpp:199:36

1: ./PeleLMeX2d.gnu.MPI.ex() [0xf1daa6]
amrex::BLBackTrace::handler(int)
/share/home/zhengjian/Pele/amrex/Src/Base/AMReX_BLBackTrace.cpp:99:15

2: ./PeleLMeX2d.gnu.MPI.ex() [0x5cc38e]
amrex::Abort(char const*) inlined at /share/home/zhengjian/Pele/PeleLMeX/Source/PeleLMDiffusion.cpp:1148:18 in PeleLM::differentialDiffusionUpdate(std::unique_ptr<PeleLM::AdvanceAdvData, std::default_deletePeleLM::AdvanceAdvData >&, std::unique_ptr<PeleLM::AdvanceDiffData, std::default_deletePeleLM::AdvanceDiffData >&)
/share/home/zhengjian/Pele/amrex/Src/Base/AMReX.H:159:19
PeleLM::differentialDiffusionUpdate(std::unique_ptr<PeleLM::AdvanceAdvData, std::default_deletePeleLM::AdvanceAdvData >&, std::unique_ptr<PeleLM::AdvanceDiffData, std::default_deletePeleLM::AdvanceDiffData >&)
/share/home/zhengjian/Pele/PeleLMeX/Source/PeleLMDiffusion.cpp:1148:18

3: ./PeleLMeX2d.gnu.MPI.ex() [0x57be7c]
PeleLM::oneSDC(int, std::unique_ptr<PeleLM::AdvanceAdvData, std::default_deletePeleLM::AdvanceAdvData >&, std::unique_ptr<PeleLM::AdvanceDiffData, std::default_deletePeleLM::AdvanceDiffData >&)
/share/home/zhengjian/Pele/PeleLMeX/Source/PeleLMAdvance.cpp:362:4

4: ./PeleLMeX2d.gnu.MPI.ex() [0x57ed07]
PeleLM::Advance(int)
/share/home/zhengjian/Pele/PeleLMeX/Source/PeleLMAdvance.cpp:172:7

5: ./PeleLMeX2d.gnu.MPI.ex() [0x5939aa]
PeleLM::Evolve()
/share/home/zhengjian/Pele/PeleLMeX/Source/PeleLMEvolve.cpp:43:18

6: ./PeleLMeX2d.gnu.MPI.ex() [0x429a3f]
main
/share/home/zhengjian/Pele/PeleLMeX/Source/main.cpp:57:51

7: /lib64/libc.so.6(__libc_start_main+0xf5) [0x2b630057f555]

8: ./PeleLMeX2d.gnu.MPI.ex() [0x4341ad]
_start at ??:?

This bug is very specific, but wondering if anyone has any thoughts on it, or have seen anything similar.

When I run on some systems (this error is not consistent across different HPC systems) with EB, the code hangs on initialisation while resetting the covered masks when I run with no AMR. I can trick it out of this by setting amr.max_level=1 and peleLM.refine_EB_max_level = 0. I don't think this would effect code performance too much since it presumably would just have an empty second entry in the vectors of MultiFabs, but it seems like a strange thing to happen. It also seems very strange that it does not happen consistently across different systems.

When the soret effect is turned on (simple, fuego), we can set up a test case, where species are "produced" at the wall due to zero gradient boundary condition. For soret, a zero gradient boundary conidition for the species is actually not correct, since the fluxes due to thermal diffusion have to be balanced by fluxes in the species profiles, leading to non zero gradient species profiles at the walls. I created a testcase (pipe with two walls on the top and the bottom with perodic bcs on the left and right) and set a temperature gradient from 300K to 700K from the bottom of the domain to the top (top wall temperature = 700K, bottom wall temperatur = 300K) and initialized the species profiles with a mixture of air and H2 (distributed equally over the domain). Over time the mass fraction of H2 increases and the mass fluxes of O2 and N2 decrease, which is physically not correct.
If you have any questions, I can also share the test case and we can discuss on this!

Thanks

Upon adding active control to my simulation case, the inlet speed kept going to zero resulting in a flash-back.

I found out by looking at PeleLM.H that m_ctrl_velMax is set to zero by default, resulting in Vnew always being equal to zero:

225     Vnew = std::max(Vnew, 0.0);
227     Vnew = std::min(Vnew, m_ctrl_velMax);

I don't have an issue anymore with using active control as I know I can just supply a value to make my case work, but perhaps something should be changed either in the TripleFlame example:

active_control.velMax = 1.0               # Optional: limit inlet velocity

Or in the code by adding a condition for line 227 of PeleLMFlowControler.cpp (slow ?) or setting an arbitrary large default value in PeleLM.H (not good practice ?).

I would be appy to submit a pull request with the corresponding changes.

Hi all,

It seems like the PremBunsen2D tutorial case fails to run with the given inputs. On the first time step, an massive streamwise velocity is estimated. Attached is part of the output. Any idea what is going on here? Thanks!

Mike

SUNDIALS initialized.
MPI initialized with 112 MPI processes
MPI initialized with thread support level 0
AMReX (22.05-27-g5d88558d2aef) initialized
Successfully read inputs file ... 

 ================= Build infos =================
 PeleLMeX    git hash: v21.10-118-g23397a2-dirty
 AMReX       git hash: 22.05-27-g5d88558d2
 PelePhysics git hash: v0.1-1007-gb0d3f35b
 AMReX-Hydro git hash: 343e20e
 ===============================================

==============================================================================
 State components 
==============================================================================
 Velocity X: 0, Velocity Y: 1 
 Density: 2
 First species: 3
 Enthalpy: 24
 Temperature: 25
 thermo. pressure: 26
 => Total number of state variables: 27
==============================================================================

 Initializing data for 2 levels 
 Initialization of Transport ... 
 Initialization of chemical reactor ... 
Creating ReactorBase instance: ReactorCvode
Initializing CVODE:
  Using atomic reductions
 Mixture fraction definition lacks fuelID: consider using peleLM.fuel_name keyword 
25 variables found in PMF file
580 data lines found in PMF file
 Making new level 0 from scratch
 Making new level 1 from scratch
 Resetting fine-covered cells mask 
==============================================================================
Typical values: 
	Velocity: 0.1 3.923852814e+179 
	Density: 0.6650455194
	Temp:    1260.890134
	H:       -218228.8977
	Y_H2: 0.0003458573634
	Y_H: 3.14226256e-05
	Y_O: 0.000241012551
	Y_O2: 0.2074301803
	Y_OH: 0.0006121461777
	Y_H2O: 0.01451381058
	Y_HO2: 5.371213803e-05
	Y_CH2: 6.546128626e-06
	Y_CH2(S): 4.950200534e-07
	Y_CH3: 0.0002058628013
	Y_CH4: 0.04688699785
	Y_CO: 0.006938410704
	Y_CO2: 0.01541563049
	Y_HCO: 9.539561314e-06
	Y_CH2O: 0.0002204527497
	Y_CH3O: 1.360707325e-05
	Y_C2H4: 0.0002558423542
	Y_C2H5: 9.233090566e-06
	Y_C2H6: 0.0003360756718
	Y_N2: 0.762207865
	Y_AR: -2.030418186e-127
==============================================================================
 Est. time step - Conv: 1.556568412e-184, divu: 1e+20
 Initial dt: 1.556568412e-187
 Initial velocity projection:   U: 0.1  V: 3.923852814e+179
 >> After initial velocity projection:   U: 0  V: 0
 Est. time step - Conv: 3e+199, divu: 1.831467954e-05
 Initial dt: 1.831467954e-08
 Initial velocity projection:   U: 0  V: 0
 >> After initial velocity projection:   U: 0  V: 0
 Doing initial pressure iteration(s) 

 ================   INITIAL ITERATION [0]   ================ 
 Est. time step - Conv: 3e+199, divu: 7.105853017e-173
 STEP [-1] - Time: 0, dt 7.105853017e-176
   SDC iter [1] 
   - oneSDC()::MACProjection()   --> Time: 0.009342319332
   - oneSDC()::ScalarAdvection() --> Time: 0.002244584262
   - oneSDC()::ScalarDiffusion() --> Time: 0.004823582247
From CVODE: At t = 0 and h = 6.01889e-195, the corrector convergence test failed repeatedly or with |h| = hmin.```
All other processors fail as well.

In chasing down something in amr-wind (a non-subcycling code), I was looking at all the stuff happening in create_constrained_umac_grown. Specifically this line and below: https://github.com/AMReX-Combustion/PeleLMeX/blob/development/Source/PeleLMeX_UMac.cpp#L270. This stuff is present in IAMR (subcycling) as well (in some similar form): https://github.com/AMReX-Fluids/IAMR/blame/development/Source/NavierStokesBase.cpp#L1159. It is not present in incflo (non subcycling).

In chatting with @cgilet, it looks like this stuff is needed for a subcycling code but not a non-subcycling code. Therefore this stuff is probably not needed for PeleLMeX? Is this a relic of transforming from LM (subcycling) to LMeX (non-subcycling)? Do @nickwimer, @drummerdoc, @esclapez have thoughts on this?

Hello!
May I ask how to add radiative heat loss in combustion simulation?

Thank you.

This is currently untested in the CI so it gets broken sometimes.

• typical values (LE)
• multi-component linear solves (LE)
• MFbased interpolator (LE)
• Fix level data layout (LE)
• New MFI iterators
• extended CI
• convergence testing: advection, diffusion, ADR (LE)
• regression tests
• Hypre interface to linear solvers (LE)
• turbulence inflow (LE)
• hook to dump_and_stop available in AmrLevel
• use TPROF on GPU to identify any issues, have scaling data (LE)
• mass balance in closed chamber dp0dt (NW)
• mixture fraction
• control max level EB refinement (LE)
• control dt when adding level
• cylindrical coordinates support
• restore incompressible option
• Add option to prune covered boxes when EB
• Time averaged data

Can the PeleLMeX be configured to incorporate a 2D temperature field from a .dat file for the Flamesheet case?

The defaults in these cases should be changed so they work out of the box.

Discussed in #309

I want to store the diffusion coefficients of each species like the mass fractions and the production rate for each species because I need them in the PeleAnalysis part to perform calculations.

I tried to see how I_R and Y(species) are stored, but I can't find the arrays where the diffusion coefficients are stored.

Could you help me add the diffusion coefficients in the output?

Thank you in advance.

• Refactor problem setup
• Enable inflow on EBs
• Add on-the-fly diagnostics: PDF, conditional mean, iso-surface, ...
• Extend CI
• Extend V&V
• Add more tutorials
• Machinery to control dt when adding levels
• Add option to prune covered boxes when EB
• Time averaged data

After AMReX-Codes/amrex#3991 (stemming from Exawind/amr-wind#886) the PeleLMeX EB_PipeFlow test is failing on the nodal projection on the first timestep: https://github.com/baperry2/PeleLMeX/actions/runs/9784946279/job/27016932023.

Despite the CI failing on this test, I've tried with a few different machines and compilers and haven't been able to reproduce the failure locally. However, turning up verbosity for the nodal projection when running locally, I see that since the change the case takes more iterations for the nodal projection to converge (10 vs 8) and there are more frequent bottom solve failures.

@marchdf @WeiqunZhang @asalmgren @jrood-nrel it looks like you all worked the issue in amr-wind. Any idea what might be happening here in LMeX?