csdms / pymt Goto Github PK

If I want to update a plugin (e.g., rafem), where do I push those changes? I found a Python 2/3 compatibility error. I'd also like to update some of the input file defaults but am not sure where that's best done. Thanks!

Thr Hydrotrend Ganges notebook hangs on the call to initialize:

hydrotrend_GQ0.initialize(config_file, config_folder)

I've only tried this on the OpenEarthscape JupyterHub, CSDMS kernel, pymt=1.3.1.

In the recent past, we've sometimes used the name "CSDMS Workbench" to refer collectively to the various modeling tools and protocols (pymt, BMI, babelizer, Landlab, etc.). If this is to be the official name, it should appear prominently in our web portal, repositories, and documentation.

I am running the Frost number model Jupyter notebook.
https://pymt.readthedocs.io/en/latest/demos/frost_number.html
Did the installation, then i go to the notebook locally.

I do get a kernel warning and click on 'proceed with this kernel'

Running the steps sequentially in Part 1.
The last step that is supposed to calculate the air frost number for the given input, does not print the output?

Similarly if I try to do it again for the new input conditions, it does not print the output of the calculation.

Dan Parsons ([email protected]) found that setting VTK output for HydroTrend caused his WMT run to fail. Here's the stack trace:

Traceback (most recent call last): 
File "/home/csdms/wmt/bin/wmt-slave.py", line 430, in main env=env) 
File "/home/csdms/wmt/bin/wmt-slave.py", line 398, in launch task.execute() 
File "/home/csdms/wmt/bin/wmt-slave.py", line 311, in execute self.run() 
File "/home/csdms/wmt/bin/wmt-slave.py", line 380, in run model.go(file='model.yaml') 
File "/home/csdms/wmt/internal/coupling/cmt/component/model.py", line 45, in go self._components[self.driver].go(self.duration) 
File "/home/csdms/wmt/internal/coupling/cmt/component/component.py", line 157, in go self.run(stop_time) 
File "/home/csdms/wmt/internal/coupling/cmt/component/component.py", line 176, in run self._events.run(stop_time) 
File "/home/csdms/wmt/internal/coupling/cmt/events/manager.py", line 34, in run event.run(self._timeline.time) 
File "/home/csdms/wmt/internal/coupling/cmt/events/printer.py", line 18, in run self._printer.write() 
File "/home/csdms/wmt/internal/coupling/cmt/portprinter/port_printer.py", line 60, in write self._printer.write(self._field) 
File "/home/csdms/wmt/internal/coupling/cmt/printers/vtk/vtu.py", line 205, in write tofile (field, file_name, **kwargs) 
File "/home/csdms/wmt/internal/coupling/cmt/printers/vtk/vtu.py", line 174, in tofile elements = get_vtu_elements (field, format=format, encoding=encoding) 
File "/home/csdms/wmt/internal/coupling/cmt/printers/vtk/vtu.py", line 124, in get_vtu_elements types = get_vtk_types (field) 
File "/home/csdms/wmt/internal/coupling/cmt/printers/vtk/vtu.py", line 106, in get_vtk_types types = [int (edge_count_to_type.get (offsets[0], VtkPolygon))] IndexError: index out of bounds

This is error is thrown for 1D output because offsets is empty; 2D output is fine.

Dan was able to work around this issue by using netCDF output instead of VTK.

This is likely low priority, since we should just use netCDF.

The pymt README should instruct potential users how to install the software, and what docs or examples or notebooks to look at to get started.

Let's use this issue to track items to add to an FAQ for pymt. Please add suggestions as comments.

cc @mcflugen @sc0tts @iovereem

The CHANGELOG.rst file is out of date. It should be updated for recent releases. It also references Landlab.

The BMI documentation lists structured_quadrilateral as a valid grid type, but it's not available in pymt. For example:

Traceback (most recent call last):
  File "fastmech_pymt_ex.py", line 25, in <module>
    m.initialize(*defaults)
  File "/home/mpiper/anaconda3/envs/wrap/lib/python3.6/site-packages/pymt/framework/bmi_bridge.py", line 200, in initialize
    self._grid[grid_id] = dataset_from_bmi_grid(self, grid_id)
  File "/home/mpiper/anaconda3/envs/wrap/lib/python3.6/site-packages/pymt/framework/bmi_ugrid.py", line 31, in dataset_from_bmi_grid
    raise ValueError("grid type not understood ({gtype})".format(gtype=grid_type))
ValueError: grid type not understood (structured_quadrilateral)

The test_find_metadata test in test_setup.py is failing on Windows because os.path.abspath returns a lowercase drive letter, "c:" which fails when compared to uppercase "C:".

Following the instructions on the installation page:
https://pymt.readthedocs.io/en/latest/installation.html

If I type the 'import pymt' command, I get a error message, that has to do with the fact that there is still Python 2.7 installation on my machine?

Here is the message:
Traceback (most recent call last):
File "", line 1, in
File "/Users/overeem/anaconda/lib/python2.7/site-packages/pymt/init.py", line 6, in
from .plugin import load_pymt_plugins
File "/Users/overeem/anaconda/lib/python2.7/site-packages/pymt/plugin.py", line 42, in
from .framework.bmi_bridge import bmi_factory
File "/Users/overeem/anaconda/lib/python2.7/site-packages/pymt/framework/bmi_bridge.py", line 10, in
from cfunits import Units
File "/Users/overeem/anaconda/lib/python2.7/site-packages/cfunits/init.py", line 35, in
from .units import Units
File "/Users/overeem/anaconda/lib/python2.7/site-packages/cfunits/units.py", line 4, in
import netCDF4
File "/Users/overeem/anaconda/lib/python2.7/site-packages/netCDF4/init.py", line 3, in
from ._netCDF4 import *
ImportError: dlopen(/Users/overeem/anaconda/lib/python2.7/site-packages/netCDF4/_netCDF4.so, 2): Library not loaded: @rpath/libcurl.4.dylib
Referenced from: /Users/overeem/anaconda/lib/python2.7/site-packages/netCDF4/_netCDF4.so
Reason: Incompatible library version: _netCDF4.so requires version 10.0.0 or later, but libcurl.4.dylib provides version 9.0.0

(1) Install from conda forge:

conda install pymt -c conda-forge

(installation completes)

Test in ipython shell:

In [1]: import pymt
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-1-a062ed831948> in <module>
----> 1 import pymt

~/Dev/Pymt/pymt/pymt/__init__.py in <module>
      1 import pkg_resources
      2 
----> 3 from ._udunits2 import UnitSystem
      4 from .model_collection import ModelCollection
      5 

ModuleNotFoundError: No module named 'pymt._udunits2'

(2) Install from source

git clone https://github.com/csdms/pymt.git

python setup.py install

(does stuff)

Test of importing pymt: same result as above.

After quitting a Python session in which pymt was imported, an error message is shown.

To reproduce:

1. Make a new conda environment and activate it.

$ conda create -n pymt -c conda-forge python=3 pymt
$ conda activate pymt

2. Start a Python session, import pymt, then kill the session, noting the error message.

$ python
Python 3.9.6 | packaged by conda-forge | (default, Jul 11 2021, 03:36:15)
[Clang 11.1.0 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pymt
>>> ^D
Error in sys.excepthook:

Original exception was:

There's no body in the message.
The message only appears in Python, not IPython.
I'm using pymt=1.3.1.

@mdpiper @mcflugen (this might be more appropriate over in the bmi repo. But I think it is better here.

I'm writing the terrainbento bmi (in progress PR here).

When I designed terrainbento I thought about how to make output easy to use. Each saved timestep writes output in a netcdf and then I have a function that makes it easy to compile these into an Xarray dataset.

But if I want to run one of terrainbento's 28 models coupled to something else with PyMT I probably only want to save the state variables once. And I probably want to save all the variables in the same Xarray dataset.

Is this possible? Are there things I need to do with my BMI to make this work?

The tests in test_ugrid_read.py rely on a set of netCDF files on a THREDDS data server hosted on the CSDMS web server. However, CSDMS recently purchased a new web server, and we didn't install THREDDS on it. We'll needs another way to access these test files.

There is a lot of duplicated code in pymt and the model_metadata package. pymt should instead use the tools from model_metadata.

Based on an impromptu usability test, the pymt installation instructions are written in a way that leads users to the "advanced" install rather than the "quick start". This is because the item "Installation" jumps out to the eye more than "Install pymt" under "Quick start". Potential solutions could be:

1. put a link to the Quick Start version as the first item under "Installation", so maybe 3 items: Quick start installation; Advanced: Stable Release; Advanced: From Source

2. Rename the "Installation" to something like "Advanced Installation"

Should be a quick fix in the docs. Or not.

The Usage section of the pymt docs is intended to give an overview of how to use pymt to configure, run, and couple models. It currently describes how to use the "primary" (my term) set of pymt functions, such as initialize, update, finalize, get_value, set_value, etc. However, to be truly useful, it should provide descriptions for all pymt functions and properties. Here's a list:

• map_to
• map_value
• regrid
• get_component_name
• get_grid_dim
• get_grid_face_node_connectivity
• get_grid_face_node_offset
• get_grid_face_nodes
• get_grid_ndim
• get_grid_nodes_per_face
• get_grid_number_of_faces
• get_grid_number_of_nodes
• get_grid_number_of_vertices
• get_grid_origin
• get_grid_rank
• get_grid_shape
• get_grid_size
• get_grid_spacing
• get_grid_type
• get_grid_x
• get_grid_y
• get_grid_z
• get_value_ptr
• get_var_grid
• get_var_grid_loc
• get_var_intent
• get_var_itemsize
• get_var_location
• get_var_nbytes
• get_var_type
• get_var_units
• quick_plot
• time_from
• time_in
• add_data
• interpolate
• reset
• update_until

Also properties:

• author
• cite_as
• contact
• datadir
• defaults
• doi
• email
• license
• parameters
• summary
• url
• version

In the pymt documentation we should discuss, at a high level, how to couple models of differing temporal and spatial scales. We should also show examples.

this would be easier than having to get the objects with gca() etc

Currently, the docs use links (often to Wikipedia pages) to define terms like class, instance, and method. It would be better to have a glossary. Terms could then be referenced in a consistent manner, definitions could still have links to external sources, and it would be easier to define CSDMS terms like model and component.

We could use the Sphinx glossary directive.

An example is the Python glossary.

Looks like pymt_rafem isn't conda-forge. Is the plan to add it? Or should we continue to use it just as part of the software stack?

I've been working adapting SimpleCrop to be PyMT compatible and was wondering how you usually dealt with temporal mismatch between model components. The SimpleCrop model (as a wrapper around the command line interface at least) operates on a yearly basis. It takes in weather and irrigation data for each day of the year as well as some cross sectional soil and plant data. It returns plant and soil data for each day of the year. How would you deal with this mismatch? I was thinking that this could be made easier by having some generic memory components to build up results in. I was thinking of something like

from simplecrop.models import Weather, Irrigation, Crop
# may need to choose a different storage strategy in many model timesteps don't fit into memory
from pymt.storage import Memory
from pymt.model_adapters import TimeAggregator


# the weather and models crop operate on a daily basis but are converted to yearly components
weather = TimeAggregator(Weather(), to = 'year', storage = Memory())
# you may specify a keep option so that only model state used by other models is kept  
irrigation = TimeAggregator(Irrigation(), to = 'year', storage = Memory(), keep = ['irrigation_volume__total'])

weather.initialize(*weather.setup())
irrigation.initialize(*irrigation.setup())

# the crop model operates on a yearly basis
crop = Crop()

crop.initialize(*crop.setup())

# run the model for five years
for time in range(5):
    weather.update()
    irrigation.update()
    crop.set_value('irrigation_volume__total', irrigation.get_value('irrigation_volume__total'))
    crop.set_value('soil_temperature__max', weather.get_value('soil_temperature__max'))
    crop.set_value('soil_temperature__min', weather.get_value('soil_temperature__min'))
    crop.set_value(
        'soil_top_surface_raditation~incoming__energy_flux',
        weather.get_value('soil_top_surface_raditation~incoming__energy_flux'))
    crop.update()

What do you think? How do you currently handle differences in temporal range and resolution?

It would be great if pymt users could quickly find out what model and data components are available for pymt. There is a list on the CSDMS wiki website, but it would be helpful to also have:

1. a list of components, with their install commands, on the readthedocs site
2. a way to query a component list from within pymt itself (e.g., pymt.components.list())

Hi guys,
I came across a cofused problem when I install sedflux model using Anaconda prompt. The solving environment was always failed to be set. And this problem is the same with other models‘ installation, such as Hydrotrend. As shown below:

(pymt) C:\Users\张明杰>conda install pymt_hydrotrend -c conda-forge
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): failed

CondaHTTPError: HTTP 000 CONNECTION FAILED for url https://conda.anaconda.org/conda-forge/win-64/repodata.json

Hope to get your reply as soon as possible.

The cmt-config program doesn't show output for installed components. For example:

$ cmt-config > wmt-config.yaml
✓ Hydrotrend

The Hydrotrend component is installed, but cmt-config doesn't show its info:

$ cat wmt-config.yaml
components: {}
host:
  environ:
    PATH: /home/csdms/anaconda3/envs/wmt-analyst/bin:/usr/bin:/bin
  hostname: siwenna.colorado.edu
  os_name: Linux
  os_release: 3.10.0-693.5.2.el7.x86_64
  os_version: '#1 SMP Fri Oct 20 20:32:50 UTC 2017'
  platform: x86_64
  prefix: /home/csdms/anaconda3/envs/wmt-analyst
  python_version: 3.6.6

@mdpiper & @mcflugen, Our team is beginning to implement BMI coupling among a number of water models for USACE, and we're exploring various approaches for moving forward.

PyMT seems like a great tool, yet I've noticed limited recent development activity in this repo and mention of moving the capabilities of PyMT into Landlab (https://csdms.colorado.edu/csdms_wiki/images/2023_CSDMS_PMESP.pdf).

Does this mean that PyMT will be deprecated?

Following instructions on the Installation Page.
Trying to install the recommended stable release.
https://pymt.readthedocs.io/en/latest/installation.html
I literally copy the commands as listed in the grey boxes.

After the $ conda install pymt, you tell the user to start a python session.
Perhaps better to add a command line that explicitly shows the command on how to start a python session?

I am having problems trying to get pymt to run in Python version > 3.9 (on Linux).

I installed the pymt stable release via conda install in a Python 3.12 environment. Importing the module does not result in an error but it's clearly not working:

Python 3.12.2 | packaged by conda-forge | (main, Feb 16 2024, 20:50:58) [GCC 12.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pymt
>>> for model in pymt.MODELS:
...     print(model)
...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'pymt' has no attribute 'MODELS'
>>>

When I do the same in a Python 3.9 environment, it seems to work fine:

Python 3.9.18 | packaged by conda-forge | (main, Dec 23 2023, 16:33:10)
[GCC 12.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pymt
>>> for model in pymt.MODELS:
...     print(model)
...
Cem
Waves
Child
Hydrotrend
FrostNumber
Ku
KuEnhanced
Avulsion
Plume
Sedflux3D
Subside

Is there a way to get this to work in Python 3.12 (or 3.11)?

Two TimeInterpolator tests are failing. The traceback:

===================================== FAILURES ======================================
__________________________ test_interp_outside_range[next] __________________________
Traceback (most recent call last):
  File "/home/mpiper/projects/pymt/tests/framework/test_timeinterp.py", line 88, in test_interp_outside_range
    assert np.isfinite(interp(4.5)) and interp(4.5) >= 4.0
AssertionError: assert (False)
 +  where False = <ufunc 'isfinite'>(array(nan))
 +    where <ufunc 'isfinite'> = np.isfinite
 +    and   array(nan) = <pymt.framework.timeinterp.TimeInterpolator object at 0x7fe5596021a0>(4.5)
_________________________ test_interp_below_range[previous] _________________________
Traceback (most recent call last):
  File "/home/mpiper/projects/pymt/tests/framework/test_timeinterp.py", line 96, in test_interp_below_range
    assert np.isfinite(interp(-0.5)) and interp(-0.5) >= 4.0
AssertionError: assert (False)
 +  where False = <ufunc 'isfinite'>(array(nan))
 +    where <ufunc 'isfinite'> = np.isfinite
 +    and   array(nan) = <pymt.framework.timeinterp.TimeInterpolator object at 0x7fe5596012a0>(-0.5)

The problem lies with the behavior of the previous and next methods of scipy.interpolate.interp1d, which TimeInterpolator uses. The issue scipy/scipy#16997 explains the old (scipy<1.9) and new (scipy>=1.9) behavior of these methods. It looks like the new behavior is expected, so we'll need to update our tests.

The HydroShare resource "Pymt Jupyter Notebooks" lists two notebook collections in folders: "bmi-walkthrough" and "pymt tutorials". The "pymt tutorials" folder includes a handful of introductory numbered tutorials, but not the complete collection of pymt tutorial notebooks that live in the GitHub repository under the "notebooks" folder. However, if a HydroShare user opens the "Pymt Jupyter Notebooks" resource with the CSDMS CUAHSI Jupyter server, the server opens with the "notebooks" folder visible and the various tutorials accessible. So a HydroShare user can access all of these, but would not know they are available until actually opening a notebook server because for some reason the "notebooks" folder is not visible in the resource listing.

There is a need for the automated testing of the pymt Jupyter Notebooks. This will help make sure the version update won't break the notebooks.

@gantian127 Do you want me to direct people to give feedback comments here? This is SO great!

• The Notebook ran as expected from Binder. Yeah!

• The kernel crashed within 10 minutes, so limit recommended use to exploring usability, and then give directions to a more stable JupyterHub server. CSDMS? or CUAHSI?

• There are no citations or references so when I save a copy, I don't have any provenance information to carry along manually or automated.

• If this is in a testing mode, perhaps add a hyperlink to this issue to direct people how you want to collect information. Or give a Google form or any kind of user feedback instructions.

• My other comments are related to extending the use for COVID-19 seasonality. That would require making local custom versions of the notebooks for local health department or school-sheds linked to the COMID. Also would require creating historical parameters for seasonality of temperature and relative humidity as the target variables of interest.

In the _BmiBap.grid_x method, an array is allocated to hold the x values of a rectilinear grid:

The indexing holds for arrays of rank 2 and shape (y, x), but not for arrays of rank 3 and shape (z, y, x). The same indexing problem occurs in _BmiBap.grid_y and _BmiBap.grid_z.

The clue is in the traceback from a rank 3 variable:

bmi-geotiff
<xarray.Rectilinear>
Dimensions:     (rank: 3)
Dimensions without coordinates: rank
Data variables:
    mesh        int64 0
    node_shape  (rank) int32 1 1623 2043
Traceback (most recent call last):
  File "/Users/mpiper/build/pymt_geotiff/examples/pymt_geotiff-pymt_ex.py", line 15, in <module>
    m.initialize(*args)
  File "/Users/mpiper/anaconda3/envs/pymt/lib/python3.9/site-packages/pymt/framework/bmi_bridge.py", line 324, in initialize
    self._grid[grid_id] = dataset_from_bmi_grid(self, grid_id)
  File "/Users/mpiper/anaconda3/envs/pymt/lib/python3.9/site-packages/pymt/framework/bmi_ugrid.py", line 318, in dataset_from_bmi_grid
    grid = Rectilinear(bmi, grid_id)
  File "/Users/mpiper/anaconda3/envs/pymt/lib/python3.9/site-packages/pymt/framework/bmi_ugrid.py", line 244, in __init__
    nodes = self.get_nodes()
  File "/Users/mpiper/anaconda3/envs/pymt/lib/python3.9/site-packages/pymt/framework/bmi_ugrid.py", line 48, in get_nodes
    data = getattr(self.bmi, "grid_" + dim_name)(self.grid_id)
  File "/Users/mpiper/anaconda3/envs/pymt/lib/python3.9/site-packages/pymt/framework/bmi_bridge.py", line 465, in grid_x
    self.bmi.get_grid_x(grid, out)
  File "/Users/mpiper/anaconda3/envs/pymt/lib/python3.9/site-packages/bmi_geotiff/bmi.py", line 319, in get_grid_x
    x[:] = self._da.x.values
ValueError: could not broadcast input array from shape (2043,) into shape (1623,)

Note the x and y dimensions are swapped.

shapely had a major release recently, and it looks like the geometry types changed, which causes pymt to fail:

 ______________________ ERROR collecting pymt/grids/map.py ______________________
Traceback (most recent call last):
  File "/Users/runner/work/pymt/pymt/pymt/grids/map.py", line 49, in <module>
    from shapely.geometry import Point, asLineString, asPoint, asPolygon
ImportError: cannot import name 'asLineString' from 'shapely.geometry' (/usr/local/miniconda/envs/test/lib/python3.8/site-packages/shapely/geometry/__init__.py)

Hello guys,
I came across a problem about using Sedflux3D model through jupyter notebook. I want to kown if there is any detail descriptions for how to use some functions and properties, such as get_grid_x, get_grid_oringin and so on.
And Is there any detail manule or deccriptions about how to operate Sedflux3D. I mean for example where i can chage parameters for the Sedflux3D. Should I chage the parameters in the file folder Sedflux3D-prameters.yaml?

Looking forward to your help!