nasa-develop / dnppy Goto Github PK

Now that dnppy has integrated travis-ci documentation building, the environment for automated testing is already in place. All that's missing is testing for the functions.

If tests are developed for dnppy that don't use local files for testing it would be super easy to integrate said testing remotely so you would know when pushes break code.

using rast_series_class.series_statistics() with "NUM" in the desired saves produces an output raster with all zero values.

The (FIRMS data product)[https://earthdata.nasa.gov/earth-observation-data/near-real-time/firms] is a MODIS derivative. This data is of interest to at least one partner organization.

The support should include fetch and extraction to geotiff, perhaps mosaicing as well.

Each module should have its own page in this wiki, with a description of the modules purpose, simple use cases, syntax examples.

The issues tracker is the easiest place for DEVELOP participants or partners to request help when they hit a snag with DEVELOP related code. Even if the issue ends up having nothing to do with dnppy, anything with the "help!" tag will receive attention, and get an answer.

time zones are not properly applied to convert datetimes from local to GMT.

Some raster functions may assume that an array of filepaths are stepping forward in time, but this isn't enforced by time_series objects when returning lists of data.

the extract_NetCDF tool in the precipitation module is not reading the appropriate output directory when the Copy Raster function is used to save the resulting tif file. It keeps saving the files in the same folder the original nc files are in.

Before the start of the summer term, existing code needs another round of code review to ensure everything works. First full release is scheduled for May 29th 2015. After this date, all development should be done on fresh branches (not the master) until stable.

The function successfully downloads files with the "nws_precip_conus_yyyymmdd.nc" file name based on date range defined by the user. However, I'm realizing it's not shooting back an error when files requested do not exist. For example, I requested a range Jan 1st 2002 through Jan 1st 2014. It downloaded all these files and it appears they are netcdf files. However, when you look at the actually ftp there is only data for 2005 onward. If I try and read in the downloaded files prior to 2005 the file format isn't recognized as a NetCDF.

Additionally, some of the "nws_precip_conus_yyyymmdd.nc" files on the ftp are double zipped as .tar.gz. However, when the fetch_MPE downloads the files they appear as just the nc file but still can't be read in. There didn't appear to be code within the fetch_MPE for checking to see if the file is zipped and if so unzipping the file.

This doesn't presently cause any big issues, so it isn't really a bug, but it should smoothly handle rasters with may layers.

several fetch functions use ftp protocols to map directories, but then call download_url to grab the file it wants, which does not follow FTP. This causes FTP servers to reject the connection after repeated attempts.

download_url needs to handle http and ftp links differently.

To-do list item.

The University of Montana hosts some global 1km 8-day ET, and LE data products, and it would be nice to have fetch,extraction,projection support for the MODIS products on their ftp server at ftp.ntsg.umt.edu/pub/MODIS/Mirror

users are on Windows7-64bit with Arcmap 10.1 installed.

Issue

We should have seen this issue coming, but we've now hit the stage where we'd like to run autmodule on all dnppy functions and classes, and this process cannot run successfully on travis-ci without giving it some way to handle import arcpy.

Potential solutions

• Do not use autmodule on functions with arcpy dependencies. Mixing manual and automatic documentation.
• Somehow fake an arcpy import for the sake of travis-ci to get around that step.
• Switch to a documentation update strategy that requires us to build locally and commit to gh-pages

Quickly discovering that many people have specific types of files they are used to reading, many of them are fixed-width format containing weather data. These specialized text files require their own reading/writing functions, but there is a need to be able to interchangably convert between formats and combine measurements from a variety of them.

We need a "text file object" that has rows (a list of lists), and headers (a list). Each of our custom text readers can be writen to output a text file object, which can be easily handled by other modules such as the time_series module.

Request on behalf of Nathan Owen,

current netcdf conversion tools throw errors on climate data from Global Climate Model Output data. Need custom fix.

The website fails to display the same way it does after a local build, though Travis-CI shows a completed update.

What it should look like

What it does look like

chunking module has not been adequately tested on real raster data.

presently, there isn't any way to group by some custom interval without multiple representation of each subset.

for example, if i want 3 day summaries, i can use time_series.make_subsets("%d",overlap_width = 1), but this produces a subset centered around every day, while i may want only a subset every 3 days with each data point only represented a single time.

A tiny change in the name of a file from fetch_landsatWELD to fetch_LandsatWELD where the L was made uppercase fails to a capture interest for version tracking, so the remove services at travis-ci is attempting to import a module that no longer exists, and fails to build the doc pages.

Yea, I noticed the fetch_MPE function wasn't written in the init script in the download module. I can easily changed this but I wasn't sure if it was left out for a reason. Looking for confirmation that it is okay to add fetch_MPE to the init

Line 14: files.append(line.replace('/',' ').split(' " ')[5]) is only set to work with directory or file path that is structured the same as the example provided, http://e4ftl01.cr.usgs.gov/

I am attempting to use the function on http://water.weather.gov/precip/p_download_new/
And I'm not getting the right string names as a result. I think this issue has to do with the 5 in Line 14.

Now that your setup.py script is working, it would be really useful to simply upload Dnppy releases to PyPi so you the install process is simplified to pip install dnppy
uploading packages to PyPi

It would also be useful in that you can have release versions available on PyPi, where future work on the repository won't affect the package on PyPi

When averaging rasters with enf_rastlist and many_stats, the many_stats function changes NoData values within extent to 0 values when it should stay as NoData values. Therefore, when I have the final averaged raster, there are a lot of 0 values that show up where it should really be NoData. This is throwing off the raster properties. Is there a way to keep NoData values as NoData?

This used scipy function (http://docs.scipy.org/doc/scipy-0.15.1/reference/generated/scipy.stats.scoreatpercentile.html) should be replaced by the much faster numpy.percentile function in the landsat.cloud_mask functions.

@djjensen

Presently used in testing

Raster data
Landsat 8 (fetching with AWS)
Landsat 7 (fetching is non-generalized)
Landsat 5 (fetching is non-generalized)
Landsat 4 (fetching is non-generalized)
MODIS - MOD09A1 (daily surface reflectance)
MODIS - MYD09A1 (daily surface reflectance)
MODIS - MOD10A1 (daily snow cover)
MODIS - MYD11A1 (daily land surface temperature)
MODIS - MOD13A1 (16 day vegetation indices)
MODIS - MYD13A1 (16 day vegetation indices)
SRTM DEM
GPM
TRMM 3B42

non-raster data
VA administrative boundaries polygons

Wishlist for automatic retrieval and testing.

raster data
Climate data record NetCDFs
ASTER DEM
VIIRS?

non-raster data
weather data?
vector data for metro areas?

presently, the scene_list.txt file makes up about 95% of the dnppy space requirement. This information isn't easily found online since it is only available for download in compressed format from AWS here:
http://landsat-pds.s3.amazonaws.com/scene_list.gz

The current solution is to simply include it in the dnppy package, but this isn't preferred.

A users experience could be dramatically improved if they could (for any given item with a bounding box of lat/lon coordinates) view that object, select a few NASA data products they are interested in seeing for certain time domain constraints and click a button to have formatted (and possibly clipped) geotiffs deposited onto their hard drives. One of the key missing links to do this is the simple translation from lat/lon on earth to the various tiling and naming conventions of all the products. This could be done with some kind of location class. This got started in commit 1e9801a on the dl-loc-class branch.

wishlist:
modis sinusoidal tile
modis polar tile
WRS2 (all landsat products)
degree arc (1 degree by 1 degree segment for SRTM)
Continental united states "CONUS" (true/false value)

when running setup for dnppy, wheel files sometimes fail to download from dnppy release assets. The cause is unknown.

Users who reported the issue are running Arcmap 10.1 with a 32 bit installation of python 2.7.

Having a custom programmed function to emulate arcmaps RasterToNumpyArray would be a big first step towards the ability to perform raster data analysis in a standalone python environment without commercial software.

Digital elevation model based inputs should be added to the solar module to assist in computation of irradiance and insolation like parameters. This would compliment the already in place ability to perform matrix operations for lat/lon inputs corresponding to a gridded raster dataset. These values are pertinent to energy balance applications.

Some similar functionality already exists in the ET module under the METRIC model, though it is not as scalable as it could be.

The presently supported use of Travis-ci is running on Legacy with sudo. This doesn't seem to be an immediate issue.

See merges #55 and #57

http://docs.travis-ci.com/user/migrating-from-legacy/?utm_source=legacy-notice&utm_medium=banner&utm_campaign=legacy-upgrade

This is a big one, that must be slowly worked towards. Nearly all of what dnppy does can be done without using arcpy, but so much of our code was built with arcpy from the start. Now that gdal is easily set up with installation of dnppy, effort should be made to keep future functions free of the arcpy module. Nearly all of the arcpy dependent functions are within the raster and landsat modules.

The documentation pages are incomplete, but the core module is online as a starting test case. (https://nasa-develop.github.io/dnppy/modules/core.html#code-help)

Full paths to function names are used by sphinx's automodule call, and while tediously accurate, it isn't very readable. It would be nice to take just the last element, the function name.

laptop on wifi seems to truncate the pip install script on download. not sure why yet.

it would be nice if dnppy could be installed with pip install dnppy. This may not be practical given the dependencies it requires, and may get complicated if arcpy remains a dependency

ET module currently uses a devleopment clone of some dnppy functions in a "dnppy_limited" folder. This was used to hand off this code to the project partner on an extremely aggressive timeline, but needs to be phased out.

Automatic download of very specific test data would be a nice thing to include

follow this git guide to make dnppy a citable reasearch codebase https://guides.github.com/activities/citable-code/

Issue addresses the following error.

module compiled against API version 9 but this version of numpy is 7

Numpy does not always seem to uninstall correctly. Gdal and other dependencies require numpy version 1.9.1, and they fail to import properly when numpy 1.7 is still present from arcmap.

This failure seems to be caused by access restrictions to delete all existing numpy files on the system. These files are sometimes in use (typically still by arcmap background processes) and cannot be deleted, so they are left there and the installation is left incomplete.

The only foolproof method discovered so far for correcting this is to manually delete the numpy folder from site-packages. This is definitely on the to-do list.

If you experience a numpy related issue during installation, navigate to your arcmap python site-packages folder and manually delete the numpy folder from site-packages and try dnppy installation again.

You can find the numpy folder in C:\Python27\ArcGIS10.3\Lib\site-packages or similar depending on your system configuration and arcmap version.