ncar / pyconform Goto Github PK

A package for transforming a NetCDF dataset into a defined format suitable for publication according to a defined publication standard.

Send questions and comments to Kevin Paul ([email protected]) or Sheri Mickelson ([email protected]).

The PyConform package is a Python-based package for converting model time-series data into MIP-conforming (i.e., standardized) time-series data. It was designed for CMIP6 specifically for NCAR's CESM CMIP6 workflow, but we attempted to design the code in a way that is general purpose. PyConform attempts to divide the standardization problem specification step into two separate pieces:

1. a specification of the standard, and
2. a specification of the conversion process.

This separate was created to allow the standard to be defined by (for example) the MIP designers and the conversion process to be defined by the model developers (i.e., scientists). For CMIP6, we used the dreqpy utility to define the standard, and the scientists then just needed to provide one-line definitions for how to convert the raw CESM data into the requested standardized output.

Currently, the main considerations that need to be made when creating definitions are the following:

1. physical units will be converted automatically, if possible according to the cf_units package,
2. the dimensions of the resulting data variable produced by the definition operation must be mappable to requested dimensions specified in the standard, and
3. special operations/computations that are not supplied with PyConform in the functions module may need to be written by hand and called explicitly in the output variable definition.

The PyConform package directly depends upon 4 main external packages:

• ASAPTools (>=0.6)
• cf-units
• dreqpy
• netCDF4-python
• ply
• python-dateutil

These dependencies imply the dependencies:

• numpy (>=1.5)
• netCDF4
• MPI
• UDUNITS2

Additionally, the entire package is designed to work with Python v2.7 and up to (but not including) Python v3.0.

The version requirements have not been rigidly tested, so earlier versions may actually work. No version requirement is made during installation, though, so problems might occur if an earlier versions of these packages have been installed.

Currently, the most up-to-date development source code is available via git from the site:

https://github.com/NCAR/PyConform

Check out the most recent stable tag. The source is available in read-only mode to everyone. Developers are welcome to update the source and submit Pull Requests via GitHub.

Installation of the PyConform package is very simple. After checking out the source from the above svn link, via:

$ git clone https://github.com/NCAR/PyConform

Enter the newly cloned directory:

$ cd PyConform

Then, run the Python setuptools setup script. On unix, this involves:

$  python setup.py install [--prefix=/path/to/install/location]

The prefix is optional, as the default prefix is typically /usr/local on linux machines. However, you must have permissions to write to the prefix location, so you may want to choose a prefix location where you have write permissions. Like most distutils installations, you can alternatively install the PyReshaper with the '--user' option, which will automatically select (and create if it does not exist) the $HOME/.local directory in which to install. To do this, type (on unix machines):

$  python setup.py install --user

This can be handy since the site-packages directory will be common for all user installs, and therefore only needs to be added to the PYTHONPATH once.

The documentation for PyConform is hosted on GitHub Pages.