Comments (32)
doutriaux1
commented on September 27, 2024
when did you create your conda env? I uploaded an updated cdms2 last night.
from cdms.
durack1
commented on September 27, 2024
@doutriaux1 here we go:
(uvcdat280) bash-3.2$ ls -ald /uvcdat280/lib/python2.7/site-packages/cdms2
drwxr-xr-x 100 duro THE-LAB\Domain Users 3400 Nov 16 15:33 /uvcdat280/lib/python2.7/site-packages/cdms2
(uvcdat280) bash-3.2$ conda update cdms2
Fetching package metadata .......
# All requested packages already installed.
# packages in environment at /uvcdat280:
#
cdms2 2.8 np111py27_2 uvcdat
from cdms.
doutriaux1
commented on September 27, 2024
please do a force
from cdms.
durack1
commented on September 27, 2024
@doutriaux1 same issue, it's a problem with cdms:
(uvcdat280) bash-3.2$ conda update cdms2 --force
Fetching package metadata .......
Solving package specifications: ..........
Package plan for installation in environment /uvcdat280:
The following NEW packages will be INSTALLED:
cdms2: 2.8-np111py27_2 uvcdat
Proceed ([y]/n)? y
Pruning extracted packages from the cache ...
[ COMPLETE ]|########################################################################################################################################################| 100%
Extracting packages ...
[ COMPLETE ]|########################################################################################################################################################| 100%
Linking packages ...
(uvcdat280) bash-3.2$ ipython
Python 2.7.12 | packaged by conda-forge | (default, Sep 8 2016, 14:41:48)
Type "copyright", "credits" or "license" for more information.
IPython 5.1.0 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object', use 'object??' for extra details.
In [1]: import cdms2 as cdm
In [2]: infile = '161122_RobertPincus_multiple_input4MIPs_radiation_RFMIP_UColorado-RFMIP-20161122_none.nc'
In [3]: f = cdm.open(infile)
---------------------------------------------------------------------------
CDMSError Traceback (most recent call last)
<ipython-input-3-da3bd29babf2> in <module>()
----> 1 f = cdm.open(infile)
/uvcdat280/lib/python2.7/site-packages/cdms2/dataset.pyc in openDataset(uri, mode, template, dods, dpath, hostObj)
357
358 # The file exists
--> 359 file1 = CdmsFile(path, "r")
360 if libcf is not None:
361 if hasattr(file1, libcf.CF_FILETYPE):
/uvcdat280/lib/python2.7/site-packages/cdms2/dataset.pyc in __init__(self, path, mode, hostObj, mpiBarrier)
1137 # Get grid information for the variable. gridkey has the form
1138 # (latname,lonname,order,maskname, abstract_class).
-> 1139 gridkey, lat, lon = var.generateGridkey(self._convention_, self.variables)
1140
1141 # If the variable is gridded, lookup the grid. If no such grid exists,
/uvcdat280/lib/python2.7/site-packages/cdms2/avariable.pyc in generateGridkey(self, convention, vardict)
239 lon, nlon = convention.getVarLonId(self, vardict)
240 if (lat is not None) and (lat is lon):
--> 241 raise CDMSError, "Axis %s is both a latitude and longitude axis! Check standard_name and/or axis attributes."%lat.id
242
243 # Check for 2D grid
CDMSError: Axis lon is both a latitude and longitude axis! Check standard_name and/or axis attributes.from cdms.
durack1
commented on September 27, 2024
@doutriaux1 curiously, in a previous (and not identical) version of the file, if the lon:units were changed from "degree_north" to "degree north" (remove the underscore) it seemed to be readable by the checker/cdms
@dnadeau4 pinging you into this thread
from cdms.
dnadeau4
commented on September 27, 2024
degree_north and degree_east are accepted and CDMS2 should work with this file. cf-conventions
I will enhanced CDMS2 to make sure your lat/lon are detected correctly
from cdms.
durack1
commented on September 27, 2024
@dnadeau4 out of curiosity, I tried to update the file to include lon:axis = "X" and similar for lat and I still get the identical error:
[durack1@oceanonly RFMIP]$ ncdump -h test.nc | more
netcdf test {
dimensions:
expt = 18 ;
level = 61 ;
layer = 60 ;
site = 100 ;
variables:
float lon(site) ;
lon:long_name = "ERA-Interim longitude" ;
lon:units = "degree_north" ;
lon:standard_name = "longitude" ;
lon:axis = "X" ;
float lat(site) ;
lat:long_name = "ERA-Interim latitude" ;
lat:units = "degree_east" ;
lat:standard_name = "latitude" ;
lat:axis = "Y" ;
float time(site) ;
time:long_name = "ERA-Interim fractional day of the year 2014" ;
time:units = "days since 2014-1-1 0:0:0" ;
time:standard_name = "time" ;
time:calendar = "gregorian" ;
float sst(site) ;
sst:title = "sea surface temperature" ;
sst:units = "K" ;
sst:long_name = "ERA-Interim sea surface temperature (= \"missing_value\" over land)" ;
sst:standard_name = "sea_surface_temperature" ;
sst:missing_value = -9.99f ;
sst:FillValue = -9.99f ;
sst:coordinates = "lon lat time" ;
And the error:
(uvcdatNightly) duri@ocean:[RFMIP]:[2858]> ipython
Python 2.7.12 |Continuum Analytics, Inc.| (default, Jul 2 2016, 17:42:40)
Type "copyright", "credits" or "license" for more information.
IPython 5.1.0 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object', use 'object??' for extra details.
In [1]: import cdms2 as cdm
In [2]: infile = 'test.nc'
In [3]: f = cdm.open(infile)
---------------------------------------------------------------------------
CDMSError Traceback (most recent call last)
<ipython-input-3-da3bd29babf2> in <module>()
----> 1 f = cdm.open(infile)
/uvcdatNightly/lib/python2.7/site-packages/cdms2/dataset.pyc in openDataset(uri, mode, template, dods, dpath, hostObj)
307
308 # The file exists
--> 309 file1 = CdmsFile(path,"r")
310 if libcf is not None:
311 if hasattr(file1, libcf.CF_FILETYPE):
/uvcdatNightly/lib/python2.7/site-packages/cdms2/dataset.pyc in __init__(self, path, mode, hostObj, mpiBarrier)
1063 # Get grid information for the variable. gridkey has the form
1064 # (latname,lonname,order,maskname, abstract_class).
-> 1065 gridkey, lat, lon = var.generateGridkey(self._convention_, self.variables)
1066
1067 # If the variable is gridded, lookup the grid. If no such grid exists,
/uvcdatNightly/lib/python2.7/site-packages/cdms2/avariable.pyc in generateGridkey(self, convention, vardict)
236 lon, nlon = convention.getVarLonId(self, vardict)
237 if (lat is not None) and (lat is lon):
--> 238 raise CDMSError, "Axis %s is both a latitude and longitude axis! Check standard_name and/or axis attributes."%lat.id
239
240 # Check for 2D grid
CDMSError: Axis lon is both a latitude and longitude axis! Check standard_name and/or axis attributes.from cdms.
durack1
commented on September 27, 2024
@dnadeau4 if you amend the lon:units = "degree north" (remove the underscore) it loads fine:
(uvcdatNightly) duro@ocean:[RFMIP]:[2858]> ipython
Python 2.7.12 |Continuum Analytics, Inc.| (default, Jul 2 2016, 17:42:40)
Type "copyright", "credits" or "license" for more information.
IPython 5.1.0 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object', use 'object??' for extra details.
In [1]: import cdms2 as cdm
In [2]: infile = 'test_lonUnitTweak.nc'
In [3]: f = cdm.open(infile)
In [4]:
And for completeness the ncdump of the top of the file:
netcdf test_lonUnitTweak {
dimensions:
expt = 18 ;
level = 61 ;
layer = 60 ;
site = 100 ;
variables:
float lon(site) ;
lon:long_name = "ERA-Interim longitude" ;
lon:standard_name = "longitude" ;
lon:axis = "X" ;
lon:units = "degree north" ;
float lat(site) ;
lat:long_name = "ERA-Interim latitude" ;
lat:units = "degree_east" ;
lat:standard_name = "latitude" ;
lat:axis = "Y" ;
from cdms.
durack1
commented on September 27, 2024
@dnadeau4 just wondering where this issue is sitting? We're hoping to get @RobertPincus's data published in the input4MIPs project, however the publisher/cdms can't deal with his data currently..
from cdms.
dnadeau4
commented on September 27, 2024
Nothing yet, I am working on issue #7. @doutriaux1 gave me a priority list check label #priority and talk with @doutriaux1.
from cdms.
durack1
commented on September 27, 2024
@sashakames with #76 it seems that @dnadeau4 has solved this issue and so this version of bug-fixed cdms will need to be pulled across so the ESGF publisher can open and publish the file (attached above). Once this PR is merged, can we prioritise this bug fix patch into ESGF?
from cdms.
dnadeau4
commented on September 27, 2024
@durack1 this is now working. The issue was the wrong units for latitude/longitude. I used nco to change them and now cdms2 works.
ncatted -O -h -a units,lon,o,c,degree_east ~/Downloads/161122_RobertPincus_multiple_input4MIPs_radiation_RFMIP_UColorado-RFMIP-20161122_none.nc
ncatted -O -h -a units,lat,o,c,degree_north ~/Downloads/161122_RobertPincus_multiple_input4MIPs_radiation_RFMIP_UColorado-RFMIP-20161122_none.ncfrom cdms.
durack1
commented on September 27, 2024
@dnadeau4 the issue wasn't incorrect units, these were changed to prevent the CF checker (which uses a much earlier version of "chat-lite") from crashing.. I confirmed the same behaviour in the 2.8 cdms release and hence this issue
from cdms.
durack1
commented on September 27, 2024
@dnadeau4 if you see the original ncdump output, these are the same as your ncatted edits above
from cdms.
dnadeau4
commented on September 27, 2024
@durack1 I was able to work with these files when I changed the units. Why is longitudes degrees_north in the original ncdump?
from cdms.
dnadeau4
commented on September 27, 2024
degree is an acceptable unit in CF-1 conventions. When you set degree north I assumed CDMS used the first word degree. But if you set longitude to degree_north the command isLatitude returns true. That is why you get the error that lat is equal to lon.
from cdms.
dnadeau4
commented on September 27, 2024
longitude needs to be set to degree_east
from cdms.
sashakames
commented on September 27, 2024
Once ESGF has CDMS imported via a CONDA python, we will have this and any other cdms2 features. Targeting the next major release --we have one this/next week but too soon for inclusion now.
…-Sasha
From: "Paul J. Durack" <[email protected]>
Reply-To: UV-CDAT/cdms <[email protected]>
Date: Sunday, January 15, 2017 at 1:38 PM
To: UV-CDAT/cdms <[email protected]>
Cc: Sasha Ames <[email protected]>, Mention <[email protected]>
Subject: Re: [UV-CDAT/cdms] File open fails when lon and lat are variables (not dimensions) (#32)
@sashakames<https://github.com/sashakames> with #76<#76> it seems that @dnadeau4<https://github.com/dnadeau4> has solved this issue and so this version of bug-fixed cdms will need to be pulled across so the ESGF publisher can open and publish the file (attached above). Once this PR is merged, can we prioritise this bug fix patch into ESGF?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<#32 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AHf2-5HQJux9qSIMouYeGhyF2_bN3KO_ks5rSpHggaJpZM4K5v2A>.
from cdms.
durack1
commented on September 27, 2024
dnadeau4 ok now I get you.. So the unit values were flipped lon:units = "degree_north" ; should have been lon:units = "degree_east" ; and likewise for lat.. I still think the software should fix (or at the very least point out) dumb user errors like this.. So checking standard_name or longname for longitude or latitude string matches rather than bombing like it did and not specifically pointing out that the units for the dimensions (longitude and latitude) are incompatible with the software (and conventions)
I wonder if there would be a way, in the case of a failure testing the units against the possible values and returning a "units invalid" error, rather than ungracefully exiting as it does. The fact that cdms can't read this file (due to the units error) is kinda worrying to me..
from cdms.
durack1
commented on September 27, 2024
@dnadeau4 you might want to create a user error tag, and assign it here.. I know @doutriaux1 likes that label alot
from cdms.
doutriaux1
commented on September 27, 2024
@dnadeau4 I think @durack1 is right here, can we catch the error and return a user understandable error here. Thanks.
from cdms.
doutriaux1
commented on September 27, 2024
@durack1 see, as I 'm getting older I become weaker (and wiser?) and I'm willing to listen to users even when they are caught in a flagrant user error
from cdms.
jypeter
commented on September 27, 2024
That's good @doutriaux1, because our new users not only do not read the docs, but they tend to assume that things will work as they expect (regardless of what the functions are actually supposed to do). They must believe in magic and fairy tales
from cdms.
durack1
commented on September 27, 2024
@doutriaux1 that robust tag may indeed get awarded one day.. My philosophy here is to fix what you can irrespective of how trivial the error is.. The fact that the units were flipped avoided the eyes of a number of folks before @dnadeau4 correctly pointed the issue out.. I think the software should have pointed this out first and suggested (or automagically implemented) a fix.
from cdms.
doutriaux1
commented on September 27, 2024
@durack1 agreed the software should point this out to the user, I am personally STRONGLY opposed to to fixing things automatically for the user. I refuse to assume that I am smarter than the user, there might be a good reason for this that we don't know of. All we can do is let the user know that we can't go forward with the data the way it is.
from cdms.
dnadeau4
commented on September 27, 2024
I guess I could try to use "standard_name" if "units" is not right for variables lat/lon and warn the user.
If there is an attribute axis='Y' then we are done! To pass the test, the variable must start wit 'lat' (mixed cases are fine).
def isLatitude(self):
id = self.id.strip().lower()
if (hasattr(self,'axis') and self.axis=='Y'): return 1
units = getattr(self,"units","").strip().lower()
if units in ["degrees_north","degree_north","degree_n","degrees_n","degreen","degreesn"]:
return 1
return (id[0:3] == 'lat') or (id in latitude_aliases) from cdms.
durack1
commented on September 27, 2024
@dnadeau4 there is always the range check approach too, so for a valid latitude this can only occupy the range between -90.0 and 90.0, and for longitude this can only be -180.0 to 360.0 any other values are an invalid variable
The preferred behaviour to me would be for these CF variables and attributes to be set if they can be found (the current searches are fine), however if anything fails, rather than aborting the read, throw a descriptive error and return the file handle. This way a user is still able to operate on the file, but won't have much (or any?) of the cdms transientVariable magical stuff to work with. In the test case above this would allow me for example to work with the file and its contents, and what I would do is then read out all the information/variables from the file into memory, and then use cdms to rewrite the file correctly
from cdms.
doutriaux1
commented on September 27, 2024
range depends on the units though. But it's probably worth adding for known units.
from cdms.
doutriaux1
commented on September 27, 2024
we can't enforce a valid range for longitudes though
from cdms.
durack1
commented on September 27, 2024
@doutriaux1 why no valid range for longitudes? These values have to sit within the range -180 to 360 degrees. That deals with the -180 to 180 grids and the 0 to 360 grids too, what other ranges are required?
from cdms.
dnadeau4
commented on September 27, 2024
@doutriaux1 Can CDMS read other projections like Mercator? If so, I cannot check the range the @durack1 asked.
from cdms.
doutriaux1
commented on September 27, 2024
@dnadeau4 as i stated range depends on units, I think mercator and such will have units in m or something like that. So you could check range based on units degrees_north or radians or such
from cdms.
Related Issues (20)
- TransientVariable indexing with numpy.int64 HOT 3
- cdscan speed HOT 6
- Masked array treated differently by numpy.where() across numpy versions HOT 3
- Error when high-temporal-resolution (3hourly) data reading in HOT 6
- cdscan KeyError: 'height' HOT 6
- FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated HOT 1
- MV2.repeat unexpected behavior
- Make mpi4py initialization optional HOT 1
- The attributes' dictionary of a cdms2 variable has a spurious 'name' key
- cdms2.setAutoBounds(wrong_param) should generate an error
- cdscan issues with poorly formed netcdf files
- cdms2 fails on opening a file with a badly encoded string attribute
- cdms2 generating wrong longitude axis topology when reading grads dat/ctl data ? HOT 3
- Building CDMS without Conda
- UnicodeDecodeError attempting to open an NCO-generated netcdf file HOT 4
- Unable to prepare environment
- Nasty segfault with erroneous netcdf-4 data
- cdms fails to load data on a specific level HOT 4
- Error: Transferred a partial file HOT 1
- Failed to fetch. Possible Reasons: CORS Network Failure URL scheme must be "http" or "https" for CORS request. HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cdms.