The test for valid_coordinate_attribute will always fail as a result of a numpy syntax error causing a ValueError to be thrown and interpreted as a test failure in cf.py. Without the try-except block, we get the following traceback:

Running Compliance Checker on the dataset from: ../datasets/netcdf/sss_rc201401.v3.0cap.nc
Traceback (most recent call last):
  File "cchecker.py", line 24, in <module>
    main()
  File "cchecker.py", line 21, in main
    args.criteria)
  File "/Users/ochang/podaac/compliance-checker/compliance_checker/runner.py", line 39, in run_checker
    score_groups = cs.run(ds, *checker_names)
  File "/Users/ochang/podaac/compliance-checker/compliance_checker/suite.py", line 87, in run
    vals = list(itertools.chain.from_iterable(map(lambda c: self._run_check(c, dsp), checks)))
  File "/Users/ochang/podaac/compliance-checker/compliance_checker/suite.py", line 87, in <lambda>
    vals = list(itertools.chain.from_iterable(map(lambda c: self._run_check(c, dsp), checks)))
  File "/Users/ochang/podaac/compliance-checker/compliance_checker/suite.py", line 31, in _run_check
    val = check_method(ds)
  File "/Users/ochang/podaac/compliance-checker/compliance_checker/cf/cf.py", line 2955, in check_missing_data
    indices = [i for i, x in enumerate(var[:]) if (var._FillValue == x or '--' == x or 'nan' == x or 'NaN' == x)]
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

Versions

Python 2.7.5, numpy 1.8.1, Mac OSX 10.9.3

LGPLv3

The command-line usage message (output of compliance-checker -h) seems to suggest several ways of specifying which tests to apply:

 optional arguments:
  --test {acdd,cf,ioos} [{acdd,cf,ioos} ...], -t {acdd,cf,ioos} [{acdd,cf,ioos} ...], --test= {acdd,cf,ioos} [{acdd,cf,ioos} ...], -t= {acdd,cf,ioos} [{acdd,cf,ioos} ...]

However, several of these variants:

$ compliance-checker -t acdd test.nc
$ compliance-checker --test acdd test.nc
$ compliance-checker --test= acdd test.nc

result in:

compliance-checker: error: argument --test/-t/--test=/-t=: invalid choice: 'test.nc' (choose from 'acdd', 'cf', 'ioos')

Maybe the usage should just say it has to be -t={acdd,cf,ioos} or --test={acdd,cf,ioos} to keep it simple.

Also, it's not quite clear how to specify multiple tests. Comma or space-separated lists don't work.

The test check_conventions_are_cf_16 in cf.py requires the Conventions global attribute to be exactly equal to the string "CF-1.6". This is indeed what the CF conventions document recommends. However, the NetCDF User's Guide allows this attribute to be a blank- or comma-separated list of convention names.

Could the check be modifed to look for "CF-1.6" as a sub-string of the Conventions attribute?

E.g. IMOS data will usually have Conventions = "CF-1.6,IMOS-1.3". We would like these files to pass the CF checker.

The check for is_reduced_horizontal_grid is not properly discriminating against datasets that aren't reduced horizontal grids.

@DanielJMaher can we reduce the size of test-data/mapping.nc, currently it is 18mb and we can't distribute something that large.

Eventually it would be nice to check for UGRID 0.9 compliance
https://github.com/ugrid-conventions/ugrid-conventions/blob/v0.9.0/ugrid-conventions.md
perhaps using some code from pyugrid
https://github.com/pyugrid/pyugrid

Is that on the roadmap?

Print tree/table with only the failures and failure messages without needing -v.

-v should print entire test suite.
-v -v should possibly print debugging info (or leave same as -v for now)

I need to explicitly install setuptools in my miniconda environment in order to run the tests. Not sure if this is a platform issue, miniconda issue - should it be added to requirements.txt?

This isn't an error but more of a question. What is actually causing the script to run soo slow when the data within the file is increased? For example, I have one file that is 325MB, which has been reduced down to 41KB by clearing out some (a lot) of the data points. All of the variables and attributes have stayed exactly the same, only the dimensions and amount of data has changed. When I run the CF check on the smaller (41KB) file, it completes in a matter of seconds. However, when I run the CF check on the larger (325MB) file, it takes forever to run (I'm on hour two of runtime right now). Does the checker actually need to pull in each individual data point within the file? Isn't it just checking variable attributes and global attributes? FYI, I'm running with both files on my local system...

I'm trying to follow the example from the README.md here
http://nbviewer.ipython.org/gist/rsignell-usgs/9b013520225da13fe455
but getting no valid checkers found ...

Do I need to import something else?

Currently, the CF checker returns failures for DSG related checks that do not apply. In our current framework, they should be skipped, not executed.

The udunitspy library is a swig wrapper (hard to install) around udunits, and udunits does not work/install on Windows properly.

@dpsnowden suggests perhaps making a pure python port (one may already exist) - I briefly looked and it seemed reasonable.

@abirger suggests perhaps a udunits webservice that can be optionally hit from the compliance-checker. We could make udunitspy an optional dependency and react to import failure in code properly.

@kknee, this fell off the plate during the last round of development because we hadn't yet solified the glider format. Once you and @kerfoot are certain that the ioos/ioosngdac file description is accurate, we need to include this in our release planning for the next phase of development. I expect that there will be other more granular issues all attached to this milestone. This issue is just there to get the ball rolling toward planning. @BeckyBaltes note that the milestone currently has no date. Up to the team to figure out where it fits in the overall release planning.

the check is iterating through the variables an extra time. Clean up would be nice.

This issue meant for discussion, including but not limited to @dpsnowden, @rsignell-usgs, @kwilcox, @DanielJMaher, @lukecampbell, @oychang ...

The scoring system currently in the compliance-checker feels not up to par. This issue attempts to explain how it got to be the way it is, and where to go from here.

Initial Design

The initial design goals were this:

• Score a dataset by multiplying the weight (low=1, medium=2, high=3) of each top-level Result by the value (0 for 0, 0.5 for any fractional, 1 for 1)
• Provide a stable upper-bound per checker, not varying per dataset (aka same upper bound no matter how many variables)

The per variable checks rely on the concept of grouping, which results in the tree-like output you see today (and needs to be clarified in documentation). Therefore, instead of something like the following:

var_temperature_units_valid passed
var_salinity_units_valid failed

You get something like this:

var
    temperature
        units
            valid  Passed
    salinity
        units
            valid  Failed

... and everything under the "var" tree counts as ONE top level Result to be added towards the total score. In this case, "var" would be 1/2, or partial, multiplied by whatever is the maximum weight observed in this tree. THIS IS COMPLICATED.

Issues

Grouped check contribution

A major downside to this is that grouped items contribute very little to the overall score. Consider the ACDD checker: most of the checks are for global attributes, then about four checks per variable, which all get grouped together.

Since the ACDD checker was first completed, this heavily guided the scoring ideas. As soon as the CF checker was taking shape, problems emerged: we do some number of global checks, then do a TON of per-variable checks, which barely contribute to the score. Simple global attribute checks should not carry more weight than the per-variable checks!

Optional checks (DSG)

Section 9 of the CF document defines DSGs. Clearly, not every dataset going through the CF checker will or should have these properties or even perform these tests (we're fixing #36 to that affect).

Therefore, there is no way to give a consistent "CF checker" upper-bound between different datasets if one is DSG and the other is not. It would have to be something like a "bolt-on" style score - "Dataset X scored 120/140 of normal CF, and 20/30 of DSG", almost treating it as if it was a second checker.

Potential Solutions

Split out CF/DSG checker

As mentioned in the optional section above, "optional" style checks can be implemented as a second checker.

Pros:

• Consistent reporting
• Could solely report DSG compliance if wanted

Cons:

• Would this mean users would have to --test=cf,cf-dsg every time?

Do away with upper bound/scoring altogether

Instead, consider a "demerit" style scoring approach. Instead of a "100/120, here's all your wrong things", just output the wrong things. For example:

CF: -6

HIGH: No cf_role detected (-3)
MEDIUM: No author attribute (-2)
LOW: Variable "temperature" is missing a comment about how important it is (-1)

Pros:

• Could possibly do away with grouping mechanism
• Still variable independent

Cons:

• Extremely negative style approach (HERE'S HOW YOU'RE WRONG, DUMMY)
• Provides no baseline other than Perfect (0)

I apologize for the length, but needed to get this all recorded. Please take a bit to digest and give some thoughts, or ask for clarification, as this is a not very well organized brain dump.

@kerfoot John's glider nc files were failing the check independent axis dimension test even though they appear to be right. Error might be in line 1340, or around.

The Cell Boundaries Check is not properly finding the length of the variable and the length of the cell boundary variable. This is causing inaccurate failures.

The enumerate function will not properly find the missing data indices. Must change to np.where

Given the following dimensions and variables, we fail:

$ ncdump -h Equirectangular sss_rc201401.v3.0cap.nc
netcdf sss_rc201401.v3.0cap {
dimensions:
    idlon = 360 ;
    idlat = 180 ;
variables:
    float idlon(idlon) ;
        idlon:long_name = "longitude" ;
        idlon:standard_name = "longitude" ;
        idlon:units = "degrees_east" ;
        idlon:comment = "midpoint of interval on uniform grid from -180 to 180 in 1 degree longitude increments" ;
        idlon:point_spacing = "1deg" ;
        idlon:axis = "X" ;
    float idlat(idlat) ;
        idlat:long_name = "latitude" ;
        idlat:standard_name = "latitude" ;
        idlat:units = "degrees_north" ;
        idlat:comment = "midpoint of interval on uniform grid from -90 to 90 in 1 degree latitude increments" ;
        idlat:point_spacing = "1deg" ;
        idlat:axis = "Y" ;
    float sss_cap(idlat, idlon) ;
        sss_cap:long_name = "Sea Surface Salinity" ;
        sss_cap:standard_name = "sea_surface_salinity" ;
        sss_cap:units = "1e-3" ;
        sss_cap:valid_min = 0.f ;
        sss_cap:valid_max = 45.f ;
        sss_cap:scale_factor = 1.f ;
        sss_cap:add_offset = 0.f ;
        sss_cap:grid_mapping = "Equirectangular" ;
        sss_cap:comment = "level-3 analysed sea surface salinity values obtained from the Combined Active Passive -CAP- algorithm with Rain Correction. Cell values are means for the temporal interval & 1degree spatial grid" ;
        sss_cap:_FillValue = -9999.f ;
        sss_cap:coordinates = "idlat idlon" ;
    char Equirectangular ;
        Equirectangular:grid_mapping_name = "latitude_longitude" ;
        Equirectangular:Standard_Parallel = 0.f ;
        Equirectangular:Longitude_of_Central_Meridian = 0.f ;
        Equirectangular:false_northing = 0.f ;
        Equirectangular:false_easting = 0.f ;
        Equirectangular:comment = "projection also referred to as Equidistant Cylindrical" ;
...

var                                    :3:    13/14 :  
    sss_cap                            :3:     8/ 9 :  
        lat_lon_correct                :3:     0/ 1 :  

This seems to be because cf.py determines if a coordinate is lat/lon by checking variable name rather than the proper way of knowing based on units.

Not properly checking for alternative coordinate variables

There is an issue installing compliance checker on Ubuntu (Version to be filled in).

Installing udunitspy, a dependency of compliance checker, is reporting the lack of a swig executable.

SWIG is a library or tool that is used to create bindings, or an interface between a python module and a dynamically linked library written in C/C++ (or another compiled language) that is usually native to the operating system.

The issue was originally reported by @mbiddle-nodc

check_geographic_region check only allows regions that are all lowercase

When a non Lat,Lon,Time,Elevation dimension is given as a coordinate, and it is not a coordinate varaible, an error was being raised. That case should pass this check as only lat,lon,time,elev dimensions are checked here.

There has recently been a long discussion on the ACDD list discussing the relative merits of including metadata as global attributes when that metadata could (or should) be computed from the actual data in the file. The classic example is including elements describing spatio-temporal bounds in global attributes in netCDF files served via THREDDS. Since THREDDS inherits the global attributes from only one file (first, last, or random) and not from the entire aggregation, attributes such as start_time, end_time often do not describe the actual time axis in the aggregated data set time.

For attributes like these, the checker should compare the global attribute with the computed quantity and note any inconsistencies. It's going to be up to us to determine which attributes fit in to this category. The ESIP wiki describes this problem and is starting to characterize the behavior of some common applications. Basically, beware of inherited global attributes.

This file loads into NetCDF-Java and is recognized as a STATION.

Exceptions

$ compliance-checker 20150210T1900_TO_20150225T1000.nc
Running Compliance Checker on the dataset from:
20150210T1900_TO_20150225T1000.nc
The following exceptions occured during the acdd checker (possibly indicate compliance checker issues):
acdd.check_lat_extents: object __array__ method not producing an array
acdd.check_time_extents: can't subtract offset-naive and offset-aware datetimes
acdd.check_lon_extents: object __array__ method not producing an array

ACDD

I'm getting ACDD failures for every variable because they don't have a coverage_content_type attribute. Should this only be tested for "data" variables? It is failing on things like crs, time, and latitude.

CF

Not correctly identifying the featureType:

                                Medium Priority                                 
--------------------------------------------------------------------------------
    Name                            :Priority: Score
all_features_are_same_type              :2:     0/0

This failure shouldn't be occurring:

--------------------------------------------------------------------------------
                  Reasoning for the failed tests given below:                   


Name                             Priority:     Score:Reasoning
--------------------------------------------------------------------------------
axis                                   :3:     3/ 4 :  
    height                             :3:     3/ 4 :  
        is_coordinate_var              :3:     0/ 1 : height is not allowed to
                                                      have an axis attr as it is
                                                      not a coordinate var

Methods for printing need to be cleaned up and merged for redundancy.

I'm trying to test compliance-checker under windows with anaconda. pip install compliance-checker is freaking out with:

HDF5_DIR environment variable not set, checking some standard locations ..
checking C:\Users\christopher.rae ...
checking /usr/local ...
checking /sw ...
checking /opt ...
checking /opt/local ...
checking /usr ...
Traceback (most recent call last):
  File "<string>", line 17, in <module>
  File  c:\users\christopher.rae\appdata\local\temp\1\pip_build_Christopher.Rae\netCDF4\setup.py", line 208, in <module>
    raise ValueError('did not find HDF5 headers')
ValueError: did not find HDF5 headers

@dpsnowden suggests it's probably related to the way PyTables or netcdf4-python were installed in conda for windows, and that @rsignell-usgs may have insight having installed related packages on windows.

Running the CF tests on some IMOS files, we get high-priority tests failing with messages like these:

units are metres, standard_name units should be m
units are Celsius, standard_name units should be K

The CF convention says "The value of the units attribute is a string that can be recognized by UNIDATA's Udunits package" (Section 3.1) and "Unless it is dimensionless, a variable with a standard_name attribute must have units which are physically equivalent (not necessarily identical) to the canonical units" (Section 3.3).

Looking at the code (cf.py), the function check_units seems to insist on the units being exactly the canonical units for any variable with a standard_name attribute, but makes some exceptions to allow some commonly used variants. These exceptions include e.g. "meter" and "meters" (US spelling), but not "metres" (British/Australian spelling).

Wouldn't it be more correct (and more flexible) to check the units using the tests of equivalence provided by UDUNITS? (e.g. udunitspy.udunits2.Unit has a method are_convertible)

if type(data_type_check) == type(scale) == type(offset): block needs ret_val append

Based on a scan of the code and the docs, it looks like the compliance checker only checks for CF standard_names, for observed properties. But that's not the only "IOOS approved" vocabulary for observed properties; see the IOOS vocabularies doc.

There also a few other, smaller vocabularies developed by IOOS and hosted on MMI that are part of IOOS SOS compliance (eg, sector, platform type); these are all or mostly found on the DescribeSensor response. It'd be good to add these to the compliance checker, too, eventually. Direct checking against MMI (or an auto-cached version of MMI vocabularies) would be nice. I may be able to help code that, later in the year, after October.

In order to incorporate compliance-checker efficiently with other command line tools, it is recommended that it returns a non-zero return value on error.

At the moment it returns 0 all the time, correct me if I'm wrong.

Here is a sample output:

$ ./cchecker.py -t tester -c strict /tmp/output.nc
Running Compliance Checker on the dataset from: /tmp/output.nc


--------------------------------------------------------------------------------
                      The dataset scored 0 out of 2 points                      
                              during the tester check                              
--------------------------------------------------------------------------------
                               Scoring Breakdown:                               


                                 High Priority                                  
--------------------------------------------------------------------------------
    Name                            :Priority: Score
fail_me                                 :3:     0/1
fail_me2                                :3:     0/1
                        No Medium priority tests present                        
--------------------------------------------------------------------------------

                         No Low priority tests present                          
--------------------------------------------------------------------------------


--------------------------------------------------------------------------------
                  Reasoning for the failed tests given below:                   


Name                             Priority:     Score:Reasoning
--------------------------------------------------------------------------------
fail_me                                :3:     0/ 1 : Attr fail_me not present
fail_me2                               :3:     0/ 1 : Attr fail_me2 not present
$ echo $?
0

I tried compliance-checker with the CO-OPS templates that have been recently distributed on ioostech_dev, and it remained quiet for a while, then gave me just one phrase - “The following tests failed:”, and exited to a command prompt. Probable cause: template file size.

In a22b488 I disabled udunits from requirements, but without it, the CF checker crashes. Needs to be reenabled, and the comment addressed (won't install on centos/rhel?).

stillusing len(var.shape) instead of ndim. stillusing names instead of units for x,y,z

Let's get a few extra pairs of eyes on the documentation and installation instructions before declaring v1.0. Call it 1.0.0-RC1 ???

I suggest @emiliom, @DanRamage, @jcothran, and @rsignell-usgs as good testers for this release.

Result object is returning a check name of compressed_data and not horz_crs_grid_mappings_projections

also a typo "valid_capping_count" instead of "valid_mapping_count"

also 'if name_again in self.grid_mapping_dict[mapping][1]:' should be looking at the variables standard_name

This check should look for all singled dimensioned variables that are not coordinate variables, but are used as a coordinate.

The checker works fine for CF and for IOOS, but runs into an issue with ACDD.

$ compliance-checker --test=acdd -v http://data.nodc.noaa.gov/thredds/dodsC/testdata/mbiddle/carocoops.sun2.buoy_2013_12_31_02.nc
Running Compliance Checker on the dataset from: http://data.nodc.noaa.gov/thredds/dodsC/testdata/mbiddle/carocoops.sun2.buoy_2013_12_31_02.nc
Traceback (most recent call last):
File "/home/matt-ubuntu/anaconda/bin/compliance-checker", line 9, in
load_entry_point('compliance-checker==1.0.0', 'console_scripts', 'compliance-checker')()
File "/home/matt-ubuntu/anaconda/bin/cchecker.py", line 21, in main
args.criteria)
File "/home/matt-ubuntu/anaconda/lib/python2.7/site-packages/compliance_checker/runner.py", line 41, in run_checker
score_groups = cs.run(ds, *checker_names)
File "/home/matt-ubuntu/anaconda/lib/python2.7/site-packages/compliance_checker/suite.py", line 98, in run
groups = self.scores(vals)
File "/home/matt-ubuntu/anaconda/lib/python2.7/site-packages/compliance_checker/suite.py", line 361, in scores
grouped = self._group_raw(raw_scores)
File "/home/matt-ubuntu/anaconda/lib/python2.7/site-packages/compliance_checker/suite.py", line 426, in _group_raw
msgs = sum(map(lambda x: x.msgs, v), [])
TypeError: can only concatenate list (not "str") to list

Consider the following excerpt:

$  ncdump -v Equirectangular sss_rc201401.v3.0cap.nc
...
    char Equirectangular ;
        Equirectangular:grid_mapping_name = "latitude_longitude" ;
        Equirectangular:Standard_Parallel = 0.f ;
        Equirectangular:Longitude_of_Central_Meridian = 0.f ;
        Equirectangular:false_northing = 0.f ;
        Equirectangular:false_easting = 0.f ;
        Equirectangular:comment = "projection also referred to as Equidistant Cylindrical" ;
...
Equirectangular = "" ;

data_types                             :3:     3/ 4 : [u'The variable
                                                      Equirectangular failed
                                                      because the datatype is
                                                      |S1']

|S1 should be an homogenous array of single characters, i.e. equivalent to numpy.character. This equivalence is further suggested by this example from the numpy documentation.

A possible solution would be to add |S1 to the list of allowed types in cf.py.

Traceback

(cc)LMC-026061 :: ~/podaac % compliance-checker --test=acdd 20140508-MODIS_A-JPL-L2P-A2014128024500.L2_LAC_GHRSST_N-v01.nc                                                                                   
Running Compliance Checker on the dataset from: 20140508-MODIS_A-JPL-L2P-A2014128024500.L2_LAC_GHRSST_N-v01.nc
Traceback (most recent call last):
  File "/Users/ochang/.virtualenvs/cc/bin/compliance-checker", line 9, in <module>
    load_entry_point('compliance-checker==0.2.0', 'console_scripts', 'compliance-checker')()
  File "/Users/ochang/.virtualenvs/cc/bin/cchecker.py", line 21, in main
    args.criteria)
  File "/Users/ochang/.virtualenvs/cc/lib/python2.7/site-packages/compliance_checker/runner.py", line 39, in run_checker
    score_groups = cs.run(ds, *checker_names)
  File "/Users/ochang/.virtualenvs/cc/lib/python2.7/site-packages/compliance_checker/suite.py", line 87, in run
    vals = list(itertools.chain.from_iterable(map(lambda c: self._run_check(c, dsp), checks)))
  File "/Users/ochang/.virtualenvs/cc/lib/python2.7/site-packages/compliance_checker/suite.py", line 87, in <lambda>
    vals = list(itertools.chain.from_iterable(map(lambda c: self._run_check(c, dsp), checks)))
  File "/Users/ochang/.virtualenvs/cc/lib/python2.7/site-packages/compliance_checker/suite.py", line 30, in _run_check
    val = check_method(ds)
  File "/Users/ochang/.virtualenvs/cc/lib/python2.7/site-packages/compliance_checker/base.py", line 189, in _dec
    ret_val = func(s, ds)
  File "/Users/ochang/.virtualenvs/cc/lib/python2.7/site-packages/compliance_checker/acdd.py", line 122, in check_var_coverage_content_type
    vars = self._get_vars(ds, 'coverage_content_type')
  File "/Users/ochang/.virtualenvs/cc/lib/python2.7/site-packages/compliance_checker/acdd.py", line 93, in _get_vars
    attrs = zip(attrs, names)
  File "/Users/ochang/.virtualenvs/cc/lib/python2.7/site-packages/compliance_checker/acdd.py", line 91, in <genexpr>
    names = (v.attrib.get('name', 'unknown') for v in vars)
  File "/Users/ochang/.virtualenvs/cc/lib/python2.7/site-packages/petulantbear/netcdf_etree.py", line 329, in attrib
    return NcVarAttrib(self)
TypeError: __init__() takes exactly 3 arguments (2 given)

Tested on fresh installs of compliance-checker==0.2.0, petulant-bear==0.1.2 installed through pip under OSX 10.9.3 and Ubuntu 14.04, python 2.7.5.

Solution

Replace v.attrib.get(...) with v.get(...) in acdd.py:91. Then everything works as expected! Seems to be related to the changes to NcVarAttrib in ioos/petulant-bear@b58277a

Added in #68 is a list of new convention strings allowed from CF-1.0 to CF-1.6. As noted by @DanielJMaher in the comments of that PR, this checker is only designed for CF-1.6. Allowing other checks gives a false impression that other CF standards will be rigorously checked (they won't be).

I'm fine with the other changes of that PR but this list needs to be reverted back to a single entity, CF-1.6.

This is a future issue.

Checkers should be done in a manner to allow independent authoring. Typical Python approach to this is namespace packages, but caused me so much grief during initial packaging that I don't really want to revisit (nor is it a perfect solution anyway).

This looks like a great alternative: https://github.com/mitsuhiko/pluginbase

Right now, checks bomb out completely when errors occur. This is fine for debugging/development, but should be captured/recorded and allowed to continue.

Errors that occur in each check suite should be output above the normal report - just the error message/what check|suite caused it in normal mode, but with verbose on (or maybe to level 2) should give the full traceback.

Looking at the readme, output from compliance-checker --help is shown, but I cannot grep help anywhere in the compliance-checker code. Where is this help text generated?