The old data.py is still around and should be removed of clawdata.py renamed to data.py.

Reduce to a single environment variable $CLAW (at most)

Replace $CLAWUTIL by $CLAW/clawutil, etc.

Requires changes in Makefile.common and all applications Makefiles. Elsewhere?

I've got a directory with a file setrun_restart.py but no setrun.py and Makefile_restart sets SETRUN_FILE=setrun_restart.py

I modified the Makefile.common to print out what setrun file it's using...

----------------------------------------------------------------------------

Make data files needed by Fortran code:

.data: $(SETRUN_FILE) $(MAKEFILE_LIST) ;
@echo Using setrun from $(SETRUN_FILE)
$(MAKE) data

data: $(MAKEFILE_LIST);
-rm -f .data
@echo Using setrun from $(SETRUN_FILE)
$(CLAW_PYTHON) $(SETRUN_FILE) $(CLAW_PKG)
touch .data

----------------------------------------------------------------------------

It works as expected if I do
$ make data -f Makefile_restart

but here's what happens if I "make .data..."

$ make .data -f Makefile_restart
Using setrun from setrun_restart.py
make data
rm -f .data
Using setrun from setrun.py
python setrun.py amrclaw
python: can't open file 'setrun.py': [Errno 2] No such file or directory

The $(MAKE) data command is to blame -- it's not using the right version of the Makefile for this recursive call. (I also have a Makefile in the directory and if I remove that then it says it doesn't know how to make data at all.)

My GNU Make book doesn't tell how to fix this. It says there's a variable $(MAKEFLAGS) that should be passed but say that this does not contain the -f flag in particular.

Hi there, since 5.9.1 we had able to install using build/installer frontend before meson overhaul. I guess that it is an unintentional bug in meson.build.

Since the installation guidelines advice is follow pip, this issue has low impact in users. Thanks.

This issue looks related with pandas-dev/pandas#54888.

The tarball with wheel generated is attached, if unzip we found two duplicates.

$ tar -xvf target.tar.gz 
clawpack-5.9.2-cp311-cp311-linux_x86_64.whl
$ unzip clawpack-5.9.2-cp311-cp311-linux_x86_64.whl 
Archive:  clawpack-5.9.2-cp311-cp311-linux_x86_64.whl
  inflating: clawpack-5.9.2.dist-info/METADATA  
  inflating: clawpack-5.9.2.dist-info/WHEEL  
  inflating: clawpack/pyclaw/classic/classic1.cpython-311-x86_64-linux-gnu.so  
  inflating: clawpack/pyclaw/classic/classic2.cpython-311-x86_64-linux-gnu.so  
  inflating: clawpack/pyclaw/classic/classic3.cpython-311-x86_64-linux-gnu.so  
  inflating: clawpack/pyclaw/classic/classic2_sw_sphere.cpython-311-x86_64-linux-gnu.so  
  inflating: clawpack/pyclaw/sharpclaw/sharpclaw1.cpython-311-x86_64-linux-gnu.so  
  inflating: clawpack/pyclaw/sharpclaw/sharpclaw2.cpython-311-x86_64-linux-gnu.so  
  inflating: clawpack/pyclaw/sharpclaw/sharpclaw3.cpython-311-x86_64-linux-gnu.so  
  inflating: clawpack/pyclaw/sharpclaw/euler_sharpclaw1.cpython-311-x86_64-linux-gnu.so  
  inflating: clawpack/pyclaw/sharpclaw/euler_tfluct1.cpython-311-x86_64-linux-gnu.so  
  inflating: clawpack/pyclaw/limiters/weno/reconstruct.cpython-311-x86_64-linux-gnu.so  
  inflating: clawpack/pyclaw/examples/shallow_sphere/sw_sphere_problem.cpython-311-x86_64-linux-gnu.so  
  inflating: clawpack/riemann/acoustics_1D_ptwise.cpython-311-x86_64-linux-gnu.so  
  inflating: clawpack/riemann/advection_1D_ptwise.cpython-311-x86_64-linux-gnu.so  
  inflating: clawpack/riemann/acoustics_2D_ptwise.cpython-311-x86_64-linux-gnu.so  
  inflating: clawpack/riemann/acoustics_1D.cpython-311-x86_64-linux-gnu.so  
  inflating: clawpack/riemann/acoustics_variable_1D.cpython-311-x86_64-linux-gnu.so  
  inflating: clawpack/riemann/advection_1D.cpython-311-x86_64-linux-gnu.so  
  inflating: clawpack/riemann/advection_color_1D.cpython-311-x86_64-linux-gnu.so  
  inflating: clawpack/riemann/burgers_1D.cpython-311-x86_64-linux-gnu.so  
  inflating: clawpack/riemann/cubic_1D.cpython-311-x86_64-linux-gnu.so  
  inflating: clawpack/riemann/traffic_1D.cpython-311-x86_64-linux-gnu.so  
  inflating: clawpack/riemann/traffic_vc_1D.cpython-311-x86_64-linux-gnu.so  
  inflating: clawpack/riemann/traffic_vc_fwave_1D.cpython-311-x86_64-linux-gnu.so  
  inflating: clawpack/riemann/traffic_vc_tracer_1D.cpython-311-x86_64-linux-gnu.so  
  inflating: clawpack/riemann/euler_with_efix_1D.cpython-311-x86_64-linux-gnu.so  
  inflating: clawpack/riemann/euler_hlle_1D.cpython-311-x86_64-linux-gnu.so  
  inflating: clawpack/riemann/mhd_roe_1D.cpython-311-x86_64-linux-gnu.so  
  inflating: clawpack/riemann/nonlinear_elasticity_fwave_1D.cpython-311-x86_64-linux-gnu.so  
  inflating: clawpack/riemann/reactive_euler_with_efix_1D.cpython-311-x86_64-linux-gnu.so  
  inflating: clawpack/riemann/shallow_hlle_1D.cpython-311-x86_64-linux-gnu.so  
  inflating: clawpack/riemann/shallow_roe_with_efix_1D.cpython-311-x86_64-linux-gnu.so  
  inflating: clawpack/riemann/shallow_bathymetry_fwave_1D.cpython-311-x86_64-linux-gnu.so  
  inflating: clawpack/riemann/shallow_roe_tracer_1D.cpython-311-x86_64-linux-gnu.so  
  inflating: clawpack/riemann/acoustics_2D.cpython-311-x86_64-linux-gnu.so  
  inflating: clawpack/riemann/acoustics_mapped_2D.cpython-311-x86_64-linux-gnu.so  
  inflating: clawpack/riemann/advection_2D.cpython-311-x86_64-linux-gnu.so  
  inflating: clawpack/riemann/burgers_2D.cpython-311-x86_64-linux-gnu.so  
  inflating: clawpack/riemann/euler_5wave_2D.cpython-311-x86_64-linux-gnu.so  
  inflating: clawpack/riemann/psystem_2D.cpython-311-x86_64-linux-gnu.so  
  inflating: clawpack/riemann/shallow_roe_with_efix_2D.cpython-311-x86_64-linux-gnu.so  
  inflating: clawpack/riemann/shallow_bathymetry_fwave_2D.cpython-311-x86_64-linux-gnu.so  
  inflating: clawpack/riemann/sw_aug_2D.cpython-311-x86_64-linux-gnu.so  
  inflating: clawpack/riemann/shallow_sphere_2D.cpython-311-x86_64-linux-gnu.so  
  inflating: clawpack/riemann/vc_acoustics_2D.cpython-311-x86_64-linux-gnu.so  
  inflating: clawpack/riemann/vc_advection_2D.cpython-311-x86_64-linux-gnu.so  
  inflating: clawpack/riemann/vc_elasticity_2D.cpython-311-x86_64-linux-gnu.so  
  inflating: clawpack/riemann/vc_acoustics_3D.cpython-311-x86_64-linux-gnu.so  
  inflating: clawpack/riemann/euler_3D.cpython-311-x86_64-linux-gnu.so  
  inflating: clawpack/riemann/burgers_3D.cpython-311-x86_64-linux-gnu.so  
  inflating: clawpack/riemann/vc_advection_3D.cpython-311-x86_64-linux-gnu.so  
  inflating: clawpack/riemann/kpp_2D.cpython-311-x86_64-linux-gnu.so  
  inflating: clawpack/riemann/shallow_hlle_2D.cpython-311-x86_64-linux-gnu.so  
  inflating: clawpack/riemann/euler_hlle_2D.cpython-311-x86_64-linux-gnu.so  
  inflating: clawpack/riemann/euler_hlle_with_walls_2D.cpython-311-x86_64-linux-gnu.so  
  inflating: clawpack/riemann/euler_mapgrid_2D.cpython-311-x86_64-linux-gnu.so  
  inflating: clawpack/riemann/euler_4wave_2D.cpython-311-x86_64-linux-gnu.so  
  inflating: clawpack/__init__.py    
  inflating: clawpack/clawutil/__init__.py  
  inflating: clawpack/clawutil/b4run.py  
  inflating: clawpack/clawutil/chardiff.py  
  inflating: clawpack/clawutil/clawcode2html.py  
  inflating: clawpack/clawutil/claw_git_status.py  
  inflating: clawpack/clawutil/data.py  
replace clawpack/clawutil/__init__.py? [y]es, [n]o, [A]ll, [N]one, [r]ename: 

target.tar.gz

Sometimesfloat that are written out by default are not precise enough. In general we should use %20.14e or something similar to use the full precision. See clawpack/amrclaw#205 for the original use case.

Since we added the clawapps repository, I now get the the following error when running clawutil/src/python/setenv.py:

Traceback (most recent call last):
File "clawutil/src/python/setenv.py", line 256, in
outfile_base=out_file_base,**project_paths))
File "clawutil/src/python/setenv.py", line 173, in write_env_files
raise NotImplementedError("Environment settings not implemented for clawapps!")
NotImplementedError: Environment settings not implemented for clawapps!

Need to update the setenv.py functionality so that it takes into account the new setup.py installation procedure. This probably needs to just know that pip install clawpack puts things in some places and other packages still need environment variable references.

When restarting, fort.gauge files will generally be overwritten. We should first move
to fort.gauge.DATETIME so all such files can be cat'ed together at the end.

This is done in the run_with_restart.py script now found in clawpack/apps/checkpt_restart/run_with_restart, see clawpack/apps#58.

Something similar can probably be incorporated into clawpack/clawutil/src/python/clawutil/runclaw.py

See discussion on clawpack/amrclaw#137.

Some of the functionality of the utils includes tests but most of it does not.

Doing 'make .exe' in a geoclaw application expands the compile step to what's below, and in amclaw you also get 3 repetitions of the the amrclaw/src/2d

gfortran -c /Users/rjl/git/clawpack/geoclaw/src/2d/shallow/utility_module.f90
-J/Users/rjl/git/clawpack/geoclaw/src/2d/shallow
-I/Users/rjl/git/clawpack/geoclaw/src/2d/shallow/
-I/Users/rjl/git/clawpack/amrclaw/src/2d/
-I/Users/rjl/git/clawpack/amrclaw/src/2d/
-I/Users/rjl/git/clawpack/amrclaw/src/2d/
-I/Users/rjl/git/clawpack/geoclaw/src/2d/shallow/
-I/Users/rjl/git/clawpack/geoclaw/src/2d/shallow/
-I/Users/rjl/git/clawpack/geoclaw/src/2d/shallow/
-I/Users/rjl/git/clawpack/geoclaw/src/2d/shallow/
-I/Users/rjl/git/clawpack/geoclaw/src/2d/shallow/
-I/Users/rjl/git/clawpack/geoclaw/src/2d/shallow/
-I/Users/rjl/git/clawpack/geoclaw/src/2d/shallow/
-I/Users/rjl/git/clawpack/geoclaw/tests/bowl-slosh/ -O2 -fopenmp -o
/Users/rjl/git/clawpack/geoclaw/src/2d/shallow/utility_module.o

Currently the direct gauge comparison tests do not work if there are multiple cores as the gauges may be written out in a random order so the comparison will not work. Need to load each gauge individually and compare each separately.

I am not exactly sure why yet but occasionally the new Makefile formats are leading to errors that look like the following:

Errors from Test Advection2DBoundaryTest
Started 2016/02/21-20:56.50
================================================================================
At line 5 of file bc2amr.f
Fortran runtime error: Recursive call to nonrecursive procedure 'bc2amr'

I have tracked the problem down to Makefiles that contain replaced source (they specify SOURCE). A make new seems to help but it's not exactly clear what about make new fixes the problem.

Modify the function check_frame to provide better error information when the test fails. See #91 for an example for the gauge tests.

I am a little confused as to the purposes of each of these targets, in particular clean. Is clean meant to clean out the local directory data, output and plots (it does not do this right now)? Is clobber supposed to both clean and remove all compiled objects (it does do this and more)? Should new really just clobber and make the executable again?

I would propose that we redefine these options as I suggested above to make it clear what exactly the roles of these phony targets are.

PR #136 introduced some improvements to runclaw and data.py that are not backwards compatible with older versions of Python. It looks like there is not a clean way to do this so we will have to add some version checking to see what version to use.

In clawutil/src/Makefile.common, the rules

OBJECTS = $(subst .F,.o, $(subst .F90,.o, $(subst .f,.o, $(subst .f90,.o, $(SOURCES))))) %.o : %.f90 ; $(CLAW_FC) -c $< $(ALL_INCLUDE) $(ALL_FFLAGS) -o $@ %.o : %.f ; $(CLAW_FC) -c $< $(ALL_INCLUDE) $(ALL_FFLAGS) -o $@

were written assuming there might be either a file fname.f or fname.f90 but not both. If both are present and only one is listed in the Makefile, the wrong one might be compiled.

This sometimes bites us when updating a file from .f to .f90 form and the old one is still present.

A try-except block was put in temporarily to support the merge of the storm surge capabilities into the main source of GeoClaw (PR clawpack/geoclaw#80). Please do not remove the try-except block until clawpack/geoclaw#80 is resolved.

Further investigate if there's a better way to organize things than what is done in #165.

project_name is undefined at line 124 and it's not clear where that should be set since project and project_name seem to be used for the same thing in different places (but not here).

What's the correct parameter in the call to check_repos_dependencies on line 122? It's called project_name in the def but here it's looping over projects.

See #165 (comment) for the reason this is occurring, but basically, the -f flag (and any other flags) does not get passed to the recursive make calls under the new target.

Currently the default (in GNU Make anyway) for FC is set to f77. This makes it so that

FC ?= gfortran

never works (although if FC is set in the environment or on the command line this will work). Something like

ifeq ($(FC), f77)
    FC = gfortran 
endif

would work to replace f77 (since we do not support it anyway) but this really should happen above any variables the user sets.

I think there's a potential problem with the way regression tests using gauges are done.

When save == True, test.check_gauges reads in the gauge data from Fortran but then writes it out again with the Python code at line
https://github.com/clawpack/clawutil/blob/acf3fc4/src/python/clawutil/test.py#L338

This has a different format (see clawpack/pyclaw#542). To run the regression test it is the latter file that is read in and then values summed to compare with the sum of the Fortran gauge output from the latest run. These might differ just due to the different precision in the Python write.

At any rate, IMHO it would be better to just copy over the gauge file from the Fortran output into the regression_data directory when save == True. This would facilitate comparing this file to the gauge output from a new run if the test fails and one is trying to debug.

Currently src/python/clawutil/claw_git_status.py raises an exception of run from a directory that is not a git clone of clawpack (e.g. a version installed from a tar file). Instead it should just print out that fact.

This was throwing an error in a regression test that had CLAW_GIT=True in the Makefile and it would be nice to encourage that in general.

Another problem I discovered with the new Makefile system: if you replace a module with a local version, e.g. via:

MODULES = \
  amr_module.f90 \

where the local version has a different value of max1d, for example, and then do make new, this module will get compiled after the other modules listed in Makefile.amr_2d whereas it is supposed to be compiled first because other modules use this one.

Actually I think it's worse than that -- the other modules when compiled will use src/2d/amr_module.mod rather than the local version unless the other modules are also copied to the local directory and also added to the Makefile.

This suggests we shouldn't allow changing library modules in the Makefile in this manner, although the user could still add additional local modules that aren't replacing those in the library and that can be compiled after the library versions.

This move would negate a lot of duplicated code and it makes more sense to build the code (if need be) in the body of the test and not the setup. Unfortunately this would break a all the tests that implement their own runTest since these functions will have to call build_executable there instead now.

It looks like Intel has decided to deprecate the previous OpenMP compiler flag -openmp in favor of -qopenmp. We should either allow the user to set the OpenMP flag via an environment variable and/or check the version of ifort and use the correct one.

@mandli or @bolliger32, could you take a look at this?

Doing make data fails for me on the current master branch of geoclaw in the directory
$CLAW/geoclaw/examples/multi-layer/plane_wave
with the error:

Traceback (most recent call last):
  File "/Users/rjl/git/clawpack/clawpack/clawutil/data.py", line 532, in write
    fname = signature(data_object.write).parameters['out_file'].default
KeyError: 'out_file'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "setrun.py", line 521, in <module>
    rundata.write()
  File "/Users/rjl/git/clawpack/clawpack/clawutil/data.py", line 534, in write
    raise ValueError(type(data_object))
ValueError: <class '__main__.QinitMultilayerData'>

As far as I can tell this worked before 50b31ce in clawutil and not afterwards, including in the v5.6.1 release.