nasa / misr-toolkit Goto Github PK
View Code? Open in Web Editor NEWan API facilitating the access of MISR standard product files
License: Other
an API facilitating the access of MISR standard product files
License: Other
Welcome to the MISR Toolkit --------------------------- The MISR Toolkit is a simplified programming interface to access MISR L1B2, L2, and MISR-HR, conventional and ancillary data products. It is an interface built upon HDF-EOS that knows about MISR data products. It has the ability to: - Specify regions to read based on geographic location and extent or the more traditional path and block range - Map between path, orbit, block, time range and geographic location - Automatically stitch, unpack and unscale MISR data while reading - Perform coordinate conversions between lat/lon, SOM x/y, block/line/sample and line/sample of a data plane, which means geolocation can be computed instantly without referring to an ancillary data set lookups - Retrieve pixel acquistion time from L1B2 product files - Read a slice of a multi-dimensional field into an 2-D data plane (eg. RetrAppMask[0][5]) - Convert MISR product files to IDL ENVI files The MISR Toolkit has been tested on Linux CentOS 7, Mac OS X 10.14.6 and Windows 10. It's core interface is C. There are also bindings for Python (2.7 and 3.x supported) and IDL. Note that Python 2.7 is end of life and deprecated in MISR Toolkit. Complete documentation and function reference --------------------------------------------- Use a browser to view online documentation: https://nasa.github.io/MISR-Toolkit/html/index.html Offline documentation packages are also available for download on the github releases page. Third Party Library Dependencies -------------------------------- The MISR Toolkit depends on the following libraries. Download links are provide for your reference, but it is preferred to use the script "download_libraries" in the source bundle "scripts" directory. Python users will still need to download NumPy and install it according to NumPy instructions. - HDF-EOS2.18v1.00 (https://observer.gsfc.nasa.gov/ftp/edhs/hdfeos/previous_releases) - HDF4.2.10 (https://support.hdfgroup.org/ftp/HDF/releases/HDF4.2.10/src) - hdf5-1.8.16 (https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-1.8/hdf5-1.8.16/src) - netcdf-4.4.0 (ftp://ftp.unidata.ucar.edu/pub/netcdf/) - jpegsrc.v6b (https://observer.gsfc.nasa.gov/ftp/edhs/hdfeos/previous_releases) - zlib-1.2.5 (https://observer.gsfc.nasa.gov/ftp/edhs/hdfeos/previous_releases) - NumPy 1.15 or later Python module (http://www.numpy.org/) Binary Installation Instructions -------------------------------- Binary installations that do not require compilation are now available for Python 3.6, IDL, and C library users on Linux, Mac OS X, and Windows. Python 2.7 binaries are also available but deprecated on Linux and Mac OS X. For Python users, the most convenient option is downloading MISR Toolkit directly from PyPI using pip. - python -m pip install -U pip - python -m pip install -U wheel numpy - python -m pip install -U MisrToolkit A downloadable python binary wheel distribution is also available on the releases page. For IDL users, a IDL Dynamically Loadable Module (DLM) distribution is available on the releases page. To use it, download the package, extract it to a convenient location, and then follow the instructions in the README to set the IDL_DLM_PATH enviornment variable or use IDL's PREF_SET to tell IDL where you extracted the package. Source Installation Instructions -------------------------------- Linux and MacOS X The recommended location for the HDF-EOS/HDF libraries and the MISR Toolkit is /usr/local/hdfeoslibs and /usr/local/Mtk-1.5.X respectively, because this location provides convenient access for multiple user accounts. It is not necessary, however, to install in /usr/local. The MISR Toolkit and HDF-EOS can be installed anywhere. Of course installing in /usr/local requires root or sudo privileges. Use the sudo commands below if you are a sudoer or simply become root before installing the HDF-EOS/HDF libraries and before the "make install" step for the MISR Toolkit. 1) Create a working directory mkdir Mtk_tmp cd Mtk_tmp 2) Extract Mtk-src-1.5.X.tar.gz tar xzvf Mtk-src-1.5.X.tar.gz (if not done already) 3) Extract Mtk testdata (substitute Mtk-testdata-1.5.X with your version) tar xzvf Mtk-testdata-1.5.X.tar.gz 4) Download HDF-EOS/HDF, JPEG and ZLIB libraries cd Mtk-src-1.5.X scripts/download_libraries <Hit "return" for defaults> 5) Install HDF-EOS/HDF, JPEG and ZLIB libraries (using the following script is necessary because it applies patches which are required for some systems, like Mac Universal Binaries, Linux and Linux64) ** The next step requires root privileges to install into /usr/local/hdfeoslibs, otherwise choose another location and disregard sudo sudo scripts/install_hdf+hdfeos <Hit "return" for defaults (/usr/local/hdfeoslibs)> 6) Build, test and install Mtk (substitute Mtk-src-1.5.X with your version) Setup your HDF/HDFEOS environment for csh: source <your-step5-path>/bin/hdfeos_env.csh for bash: source <your-step5-path>/bin/hdfeos_env.sh Set MTK_INSTALLDIR environment variable (ex. /usr/local/Mtk-1.5.X) for csh: setenv MTK_INSTALLDIR <your-path>/Mtk-1.5.X for bash: export MTK_INSTALLDIR=<your-path>/Mtk-1.5.X Set IDL_DIR environment variable (if applicable) for csh: setenv IDL_DIR <path-to-idl>/harris/idl for bash: export IDL_DIR=<path-to-idl>/harris/idl To build, test and install (choose which libraries to build) Everything - C, command-line utilities, IDL and Python make make testall (If you have Mtk testdata installed and would like to test C, IDL and Python) ** The next step requires root privileges to install into /usr/local/Mtk-1.5.X, otherwise choose another location and disregard sudo sudo MTK_INSTALLDIR=$MTK_INSTALLDIR make install Or - C library and command-line utilites make lib make cmdutil make test (If you have Mtk testdata installed and would like to test only C routines) ** The next step requires root privileges to install into /usr/local/Mtk-1.5.X, otherwise choose another location and disregard sudo sudo make install Or - IDL library make idl make testidl (If you have Mtk testdata installed and would like to test only IDL routines - requires user interaction) ** The next step requires root privileges to install into /usr/local/Mtk-1.5.X, otherwise choose another location and disregard sudo sudo make install Or - Python library make python make testpython (If you have Mtk testdata installed and would like to test only Python routines) ** The next step requires root privileges to install into /usr/local/Mtk-1.5.X, otherwise choose another location and disregard sudo sudo make install Other build targets and options make help (for other build targets) make clean (to clean everything) If the build complains about .d files ( This occurs when the HDF-EOS environment is not set) scripts/rmdepend.sh (to remove dependency files) make 7) To use Mtk The HDF/HDFEOS environment is already set above Setup your Mtk environment (pick any): For csh: source $MTK_INSTALLDIR/bin/Mtk_c_env.csh for C source $MTK_INSTALLDIR/bin/Mtk_idl_env.csh for IDL source $MTK_INSTALLDIR/bin/Mtk_python_env.csh for Python (Don't need to do if installed in site-packages (see step 8)) For bash: . $MTK_INSTALLDIR/bin/Mtk_c_env.sh for C . $MTK_INSTALLDIR/bin/Mtk_idl_env.sh for IDL . $MTK_INSTALLDIR/bin/Mtk_python_env.sh for Python (Don't need to do if installed in site-packages (see step 8)) For C examples: $MTK_INSTALLDIR/examples/C For IDL examples: $MTK_INSTALLDIR/examples/idl For Python examples: $MTK_INSTALLDIR/examples/python For C command-line utilities source code: Mtk-src-1.5.X/src For IDL tests source code: Mtk-src-1.5.X/wrappers/idl 8) Optional - You may want to install the MisrToolkit into you Python site-packages directory cd Mtk-src-1.5.X/wrappers/python sudo python setup.py install setenv LD_LIBRARY_PATH $MTK_INSTALLDIR/lib 9) Optional - After installing the Mtk_tmp directory, it's contents is not needed, unless for reference and may be removed cd ../.. rm -rf Mtk_tmp Windows See win64/README.txt in the source or binary bundle. Known Problems -------------- The IDL routine MTK_FIND_FILELIST() does not work properly. By default OS X swallows click-events on change of focus. If IDL test windows on MacOS X 10.14 don't respond to clicks, do this command "defaults write org.x.x11 wmclickthrough -bool true" Then, once at a shell prompt, restart X11/XQuartz and retry "make testidl". IDL is not compatible with XQuartz versions greater than 2.7.9 If IDL tests return an error of "Error: attempt to add non-widget child "dsm" to parent "idl" which supports only widgets"" then you can either downgrade to XQuartz 2.7.9 or follow the procedure described at goo.gl/RvXOXy to add /opt/X11/lib/flat_namespace to the DYLD_LIBRARY_PATH set by your idl launcher script (e.g. /Applications/harris/idl87/bin/idl)
I have some difficulties in using this MISR-Toolkit, please give me a help!
I acquire the latitude/longitude from mapinfo.creat_latlon and pixel values from the .data() , but I can't convert it to a GEOTiff file in WGS84 coordinates.
I have tried to use GDAL's Geotransform, but I cannot perform transform because the latitude and longitude obtained are irregular.
In the IDL Function Summary, on the main index page (Mtk-1.5.0/MISR-Toolkit-1.5.0/doc/IDL_HTML_DOCS/IDL_MTK_Index.html), the third function in the FileQuery
section is spelled out as MTK_FILE_VERISON()
instead of MTK_FILE_VERSION()
.
Reported by Michel Verstraete:
IDL> filename = root + 'MISR_AM1_TC_CLASSIFIERS_P168_O068050_F07_0012.hdf'
IDL> fieldname = 'CloudMotionCrossTrack'
IDL> status = MTK_FIELDATTR_LIST(filename, fieldname, attrcnt, attrlist)
IDL> attrcnt
0
IDL> attrlist
Block_size.resolution_x
Block_size.resolution_y
Block_size.size_x
Block_size.size_y
Whenever the MtkInstaller-1.5.0.pkg package completes the installation of Mtk successfully on a Mac, it currently leaves the two files 'idl_MisrToolkit.dlm' and 'idl_MisrToolkit.so' in the /Applications/Mtk-1.5.0/ subdirectory, where IDL will not find them by default. It would be much more useful to copy or move those two files in the folder where IDL expects to find all DLMs, or to modify the DLM_PATH of IDL to point to this folder. If neither action can be easily implemented (or is not safe for an external script to do), then a message to this effect needs to be provided so the user understands that one more step needs to be done manually. - Michel.
When executing the MtkInstaller-1.5.0.pkg package, the progress bar in the window that opens while the process is on-going moves quickly initially, and then remains stuck for between 15 and 20 min (on my computer), perhaps due to the time required to download external libraries through a slow Internet connection. If that progress bar cannot be made to move (even slowly) while this happens, it would be useful to include a message indicating that this process can take that much time, to avoid users killing the process out of impatience. - Michel.
The Mtk function MtkFindFileList
does not work when trying to locate AGP files: the names of those files do not feature an orbit number, and the C code does not foresee the possibility of not specifying an orbit, as far as I can see... Here is what happens in IDL:
First, make sure the file exists:
IDL> res = FILE_SEARCH("/Users/michel/Codes/Julia/JMtk15/data/MISR/*_AGP_*")
IDL> res
/Users/michel/Codes/Julia/JMtk15/data/MISR/MISR_AM1_AGP_P168_F01_24.hdf
Then try to find it:
IDL> search_dir = "/Users/michel/Codes/Julia/JMtk15/data/MISR/"
IDL> product = "AGP"
IDL> path = "168"
IDL> version = "F01_24"
IDL> status = MTK_FIND_FILELIST(search_dir, product, camera, path, orbit, version, cnt, files)
% MTK_FIND_FILELIST: String expression required in this context: CAMERA.
% Execution halted at: $MAIN$
IDL> camera = ""
IDL> status = MTK_FIND_FILELIST(search_dir, product, camera, path, orbit, version, cnt, files)
% MTK_FIND_FILELIST: String expression required in this context: ORBIT.
% Execution halted at: $MAIN$
IDL> orbit = ""
IDL> status = MTK_FIND_FILELIST(search_dir, product, camera, path, orbit, version, cnt, files)
IDL> files
% Attempt to call undefined procedure: 'FILES'.
% Execution halted at: $MAIN$
IDL> PRINT, files
% PRINT: Variable is undefined: FILES.
% Execution halted at: $MAIN$
So the function requires all inputs to be provided explicitly, but does not locate the desired file because the latter does not include an orbit number. The code should probably be modified in such a way that whenever the product type is AGP, whatever value is given to camera and orbit is ignored.
The same problem occurs when calling the C function directly from within Julia, so this is not an IDL-specific problem.
Failures in the install_hdf+hdfeos script are easily missed because the script does not stop on errors.
Recommend adding 'set -e' at the top of the script to automatically exit on errors.
Here is another slight inconsistency in Mtk: When calling MtkFileType
to request the file type of either GRP ELLIPSOID or TERRAIN files, the answer includes the "_GM" (Global Mode) identifier, but when doing the same for an RCCM file, that suffix is absent, even though it is part of the file name. I know there is no Local Mode version of the RCCM file, but when reconstructing a file name from its parts, there must be additional logic to add this suffix when an RCCM file name is generated.
For instance, function MtkFindFileList
to find RCCM files and function MtkMakeFilename
to construct the name of a RCCM file both require to specify the file type as GRP_RCCM_GM
, not as GRP_RCCM
, and yield incorrect results otherwise.
Python built against nurses 6 segfaults at misr_init.c:56, assigning nline
to the global nl
.
python -c "import MisrToolkit as mtk; mtk.MtkRegion(50,100,100)"
Digging around in GDB, the symbol "nl" is defined twice:
(gdb) info address nl
Symbol "nl" is at 0x2aaaaacee6b0 in a file compiled without debugging.
(gdb) info function nl
All functions matching regular expression "nl":
...
0x00002aaaaacee6b0 nl
...
(gdb) info variable nl
All variables matching regular expression "nl":
...
0x00002aaaad0ef1d4 nl
...
The segfault is the result of trying to write to the address of a function. This can be confirmed by LD_PRELOAD
ing MisrToolkit.so
. The dynamic linker chooses the variable definition first and the segfault is avoided (although if anything calls nl
the function, things will, of course, segfault). The function nl
is defined in ncurses 6:
nm -D libncurses.so.6.0 | grep nl
00000000000216b0 T nl
Changing nl
to a different name fixes the segfault entirely, which I have tested.
For clarity and completeness, copy the README, CHANGELOG and LICENSE files to the appropriate /Applications/Mtk-1.5.0/ subdirectory when the Mtk is installed on a Mac using the current MtkInstaller-1.5.0.pkg package: currently, those files are only 'visible' when downloading the source code. - Michel.
The following URLs in scripts/download_libraries are no longer valid:
curl -L -O https://observer.gsfc.nasa.gov/ftp/edhs/hdfeos/previous_releases/zlib-1.2.5.tar.gz
curl -L -O https://observer.gsfc.nasa.gov/ftp/edhs/hdfeos/previous_releases/jpegsrc.v6b.tar.gz
curl -L -O https://observer.gsfc.nasa.gov/ftp/edhs/hdfeos/previous_releases/HDF-EOS2.18v1.00.tar.Z
HDF-EOS libraries are moved to the Earthdata Wiki:
https://wiki.earthdata.nasa.gov/display/DAS/Toolkit+Downloads
It is unclear how to get direct links from the Earthdata Wiki that work with curl.
In the IDL documentation of function MTK_RAD_TO_DD
(Mtk-1.5.0/MISR-Toolkit-1.5.0/doc/IDL_HTML_DOCS/mtk_rad_to_dd.html), the arguments in the first instance of the call to this function (in the I/O section) have been inverted, though they are correct in the subsequent Examples section.
Replace MTK_RAD_TO_DD( dd, rad )
by MTK_RAD_TO_DD( rad, dd )
.
The IDL documentation for function MTK_RAD_TO_DD
states that
IDL> radians = 1.1440634
IDL> status = MTK_RAD_TO_DD(radians, decdeg)
IDL> PRINT, decdeg
65.550002
IDL> PRINT, decdeg, FORMAT = '(D18.12)'
65.550001653340
where the second PRINT
statement prints the same result with more decimals. Yet, calling the corresponding C function MtkRadToDd
from Julia with the same input yields
julia> decdeg = jMtkRadToDd_tst_1()
65.55000431538731
These two results do not differ by much, but I would not have expected such a difference given that both computations are presumably conducted in double precision (double
or Float64
). It may be worth checking that the IDL version of this function either calls the C function with the correct argument or implements the computation in double precision.
HDFView 3.1.4 reports that field "Leaf_Area_Index_Best_Estimate" of the file "MISR_AM1_AS_LAND_P168_O068050_F08_0023.nc" has the following 6 attributes:
DIMENSION_LIST
_FillValue
coordinates
long_name
standard_name
units
However, the Mtk function 'MtkFieldAttrList' (called from Julia) reports only 5 attributes, in a different order:
julia> attrlist
5-element Vector{String}:
"_FillValue"
"coordinates"
"units"
"standard_name"
"long_name"
I have never noticed that Julia modifies the order of objects returned by C, so I suspect that the C function itself returns those attributes in that order. This raises two questions:
4.1. Why is HDFView reporting more attributes than Mtk? Are some of those attributes "hidden" for certain purposes?
4.2. Why is the order of the attributes different in the HDFView and C lists? The order may not matter if one looks for an attribute by name, but it would lead to errors if one relied on the order reported by HDFView to retrieve an attribute value with the C functions on the basis of its rank in the list.
There is a slight inconsistency in the generation of AGP files (version F01_24): All file attributes are continuous strings (without blank spaces inserted), except for the penultimate item, which is set to "End Block" instead of the expected "End_Block":
IDL> filename = "/Users/michel/MISR_HR/Input/AGP/MISR_AM1_AGP_P005_F01_24.hdf"
IDL> status = MTK_FILEATTR_LIST(filename, attrcnt, attrlist)
IDL> attrlist[25:26]
Start_block
End block
This appears to be the case for all 180 AGP files. If those are going to be updated as part of the re-processing, it might be opportune to correct this and enhance the internal consistency of those file attributes.
Mtk function MtkFileBlockMetaList
does not work with MISR AS_LAND (and probably AEROSOLS) files, because it always assumes the files to be in HDF format:
IDL> filename = root + 'MISR_AM1_AS_LAND_P168_O068050_F08_0023.nc'
IDL> status = MTK_FILE_BLOCK_META_LIST(filename, nblockmeta, blockmetalist)
% MTK_FILE_BLOCK_META_LIST: HDF HDFOPEN FAILED
This error also arises when calling the C function from Julia. If such NetCDF files do not have block metadata information, I would have expected the function to return quietly, or perhaps to issue a warning, but not to cause an error condition that stops IDL or Julia processing. And if they do include such metadata, the function should be updated to handle those files.
The MtkInstaller-1.5.0.pkg package apparently looks for the IDL executable in the folder where that software product is installed when the user buys a license for that software alone. However, if one acquires (currently from Harris) the combined ENVI/IDL package, then the IDL executable is located in a subdirectory of the ENVI installation. The Mtk installation script does not find IDL and does not generate the binary files 'idl_MisrToolkit.dlm' and 'idl_MisrToolkit.so'. The suggestion is (1) if possible, to modify the script to look in at least those two places for the IDL executable (or search for it dynamically), and (2) to issue an error message if IDL is not found, so that the user knows that the installation is not complete, although the script closes by announcing successful completion. - Michel.
When applied to a GP_GMP data file (specifically MISR_AM1_GP_GMP_P168_O068050_F03_0013.hdf), function MTK_FILE_COREMETADATA_QUERY causes IDL to crash and quit, while calling the corresponding C function MtkFileCoreMetaDataQuery from Julia (on the same file) causes a segmentation fault!
The function does work correctly with other HDF files (I have tested it with AGP, RCCM, ELLIPSOID, TERRESTRIAL, and TC_CLOUD), as well as with the AS_LAND NetCDF file. If this function is not supposed to return useful information for GP_GMP files, I would expect it to check the nature of the input file and return a warning or an error message, rather than crashing...
A similar discrepancy occurs with this function too:
IDL> radians = 1.14406333
IDL> status = MTK_RAD_TO_DEG_MIN_SEC(radians, degrees, minutes, seconds)
IDL> PRINT, degrees, minutes, seconds
65 33 0.0059520230
while calling the C function directly from Julia, with the same inut yields
julia> degrees, minutes, seconds = jMtkRadToDegMinSec_tst_1()
(65, 33, 0.0010968595743179321)
Hence, same suggestion: Check that the IDL version of this function either calls the C function with the correct (un-truncated) argument or implements the computation in double precision.
Function MtkMakeFilename
aims to generate the standard filename of a MISR product, given the necessary inputs (specifically, directory, product, camera, path, orbit, version). This works fine for HDF files, but not for NetCDF files: the filename is actually correct, but the file extension is systematically set to .hdf
instead of .nc
. This used to be correct, but AS_LAND (and AEROSOLS) files have been formatted as NetCDF files for quite some time now.
This function should be updated to handle both the older and the newer formats, based on the file version identifier.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.