Ingest from nexrad from thredds (opendap) about pyart HOT 13 CLOSED

@scollis Where can I find information on the pytds module? Google finds modules which are related to MSSQL which don't seem to be correct.

I knew thredds was serving NEXRAD Level II but it was in raw msg 31 or 1 format, if they also have netCDF4 that would be great!

from pyart.

Sean Arms, Twitter @lesserwhirls
https://github.com/lesserwhirls
[email protected]

On Jul 17, 2013, at 9:11 AM, Jonathan J. Helmus [email protected] wrote:

@scollis Where can I find information on the pytds module? Google finds modules which are related to MSSQL which don't seem to be correct.

I knew thredds was serving NEXRAD Level II but it was in raw msg 31 or 1 format, if they also have netCDF4 that would be great!

—
Reply to this email directly or view it on GitHub.

from pyart.

Hello!

pytds is about one month old and is simply a place we are stuffing some tds related helper functions for our "THREDDS Data Server (TDS) - Using Python to Access Data" workshop next week here in Boulder, CO. I've been working on a TDS catalog class in python and have it to a point where I can pass it a catalog.xml URL and find all the datasets and their associated access urls under that root.

Cheers!

Sean

from pyart.

Note that you don't need the pytds module. The previous commit of that notebook does not use pytds at all:

https://raw.github.com/Unidata/tds-python-workshop/0408bddbd7d0bf33ebe2e0a6d82da4a00ce51b50/radar_level2.ipynb

Cheers,

Sean

from pyart.

Sean

Do you have any issue if we use some of the code from pytds or the notebook in Py-ART? We are releasing under a BSD license.

from pyart.

I don't think so. Let me check with the others working on the helper functions to see, just to make sure.

from pyart.

We are thinking of using the MIT license, as that is what netCDF-Java and netCDF-C use.

http://www.unidata.ucar.edu/blogs/developer/en/entry/new_license_for_netcdf_java

The text in the blog post is likely the same text we will use, with an updated year in the copyright at the beginning.

Sean

from pyart.

Hi All.

Jon, we chatted at scipy in Austin. Just a heads up, I think we are probably going to change the name "pytds" because it is already taken. And I am hoping to have an MIT license on the project ASAP. And after workshop (next week), we are going to work to make this module stand alone rather than associated with the notebooks. Stay tuned!

from pyart.

@scollis The files being served on the thredds OpenDAP server don't work nicely with the python-netcdf4 module because the field variables have a vector missing_value:

url = 'http://thredds.ucar.edu/thredds/dodsC/nexrad/level2/KATX/20130717/Level2_KATX_20130717_1950.ar2v'
import netCDF4
dataset = netCDF4.Dataset(url)
var = dataset.variables['Reflectivity_HI']
var[0,0,:]
>>> ...
>>> AttributeError: 'bool' object has no attribute 'any'

var.missing_value      # this is were the problems come from
>>>array([1, 0], dtype=int8)

# but we can turn off the scaling and masking...
var.set_auto_maskandscale(False)
var[0,0,:]
>>> array([87, 81,  0, ...,  0,  0,  0], dtype=int8)

For anyone not interested in why this is occurring of if it is breaking a standard skip the next two paragraphs.

I'm betting this is because the server is generating the server file using the same code which runs toolUI and the NetCDF Java library (http://www.unidata.ucar.edu/software/netcdf-java/). Using toolUI to convert NCDC NEXRAD level II files results in very similar looking NetCDF files with a vector missing_value attribute.

The [1, 0] missing values are in many ways correct, in Archive Level II NEXRAD files radar moment data uses 0 to indicate a signal below threshold and 1 to indicate range folded data (footnote 21 on page 3-99 in the Interface Control Document for the RDA/RPG Open Build 13.0, 2620002M). But is this allowed in NetCDF files? The NetCDF User's guide (NUG), http://www.unidata.ucar.edu/software/netcdf/docs/netcdf.html#Attribute-Conventions, says yes. The NetCDF Climate and Forecast (CF) Metadata Conventions seems to suggest this is a no-no in the section on missing data (http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.6/cf-conventions.html#missing-data) but is not clear. Now the file being served doesn't claim to follow CF conventions (dataset.Conventions is '_Coordinates') so I see no reason to call this as a non-conforming netCDF file.

Getting back to the task at hand, how can we ingest these files into Py-ART? I see four options:

1 Read these using python-netcdf4 and scale and mask the data manually after setting set_auto_maskandscale(False).

This doesn't introduce any new dependencies into Py-ART, but will only work if python-netcdf4 was compiled with OPeNDAP support (--enable-dap) which requires libcurl (which is quite common). It also mean writing a bit more code to do the masking and scaling.

2 Use Pydap (http://www.pydap.org/) to access the files, and still do the masking and scaling manually.

This introduces a number of new dependencies into Py-ART (Pydap, httplib2, Genshi, Paste, PasteDeploy, PasteScript) but they all appear(?) to be Python only modules. python-netcdf4 doesn't need to be compiled in any special way. We would still need to write the scaling and masking code as Pydap doesn't do this for us.

3 See if we can get python-netcdf4 to work with multiple variables which have missing_value.

I have know idea how hard this would be or if such a patch would be accepted by the python-netcdf4 package, but it it was supported it would make reading these files easy.

4 Get the thredds server to set only a single missing_value in the moment variables.

I'm guessing this would be a no go as the two missing values are technically correct.

Scott or anyone else have an opinion on which direction we should proceed? I'm leaning toward 1 since it is the most obtainable, but would be open to other suggestions.

Also it should be noted that I my nexrad_reader branch (https://github.com/jjhelmus/pyart/tree/nexrad_reader) should be able to read the ar2v files which can be downloaded from the thredds server using HTTP with a little more work. The message blocks just need to be decompressed during reading.

from pyart.

Ok I might have made performing the scaling, offset, and masking sound too difficult, in fact is was very easy.

The thredds_ingest branch (https://github.com/jjhelmus/pyart/tree/thredds_ingest) contains some initial code for getting a radar object from a thredds server url. Use as follows:

import pyart
from pyart.io import thredds
import matplotlib.pyplot as plt

url = 'http://thredds.ucar.edu/thredds/dodsC/nexrad/level2/KATX/20130717/Level2_KATX_20130717_1950.ar2v'
radar = thredds.read_nexrad_thredds_url(url)

display = pyart.graph.RadarDisplay(radar)
fig = plt.figure()
ax = fig.add_subplot(111)
display.plot_ppi('reflectivity', 0, ax=ax, vmin=-35, vmax=80)
fig.savefig('nexrad_ppi.png')

Result:

Looks a little too hot, I might be scaling the moments incorrectly, either that or Seattle has a good amount of rain. I'll clean up the code, make it possible to retrieve the normal resolution data, and add tests before merging into the master.

from pyart.

BTW, Sean made an NWS colormap for radar data: http://nbviewer.ipython.org/urls/raw.github.com/Unidata/tds-python-workshop/master/radar_level2.ipynb

from pyart.

A more or less complete reader for the NEXRAD Level II data served from the Thredds server is in my thredds_ingest branch, https://github.com/jjhelmus/pyart/tree/thredds_ingest. It also works with the NetCDF files creating using Unidata's NetCDF Java library (toolsUI) hence the new name CDM (common data model).

I'm merging this into my nexrad_reader branch with plans of merging that into Py-ART master once I finish the enhancements for reading NEXRAD Archive Level II files from NCDC and UCAR Thredds/Motherload are complete.

from pyart.

Pull Request #76 added support for ingest of NEXRAD Level II data from the THREDDS server as well as NCDC. I'll leave the parsing of the xml files, etc. to determine the latest radar file to grab to pytdr (or the new name) as such functionality would be outside Py-ART's scope. Closing.

from pyart.