nasa / misr-toolkit Goto Github PK

I have some difficulties in using this MISR-Toolkit, please give me ａ help！

I acquire the latitude／longitude from mapinfo.creat_latlon and pixel values from the .data() , but I can't convert it to a GEOTiff file in WGS84 coordinates．

I have tried to use GDAL's Geotransform, but I cannot perform transform because the latitude and longitude obtained are irregular.

In the IDL Function Summary, on the main index page (Mtk-1.5.0/MISR-Toolkit-1.5.0/doc/IDL_HTML_DOCS/IDL_MTK_Index.html), the third function in the FileQuery section is spelled out as MTK_FILE_VERISON() instead of MTK_FILE_VERSION().

Reported by Michel Verstraete:

IDL> filename = root + 'MISR_AM1_TC_CLASSIFIERS_P168_O068050_F07_0012.hdf'
IDL> fieldname = 'CloudMotionCrossTrack'
IDL> status = MTK_FIELDATTR_LIST(filename, fieldname, attrcnt, attrlist)
IDL> attrcnt
           0
IDL> attrlist
Block_size.resolution_x
Block_size.resolution_y
Block_size.size_x
Block_size.size_y

Whenever the MtkInstaller-1.5.0.pkg package completes the installation of Mtk successfully on a Mac, it currently leaves the two files 'idl_MisrToolkit.dlm' and 'idl_MisrToolkit.so' in the /Applications/Mtk-1.5.0/ subdirectory, where IDL will not find them by default. It would be much more useful to copy or move those two files in the folder where IDL expects to find all DLMs, or to modify the DLM_PATH of IDL to point to this folder. If neither action can be easily implemented (or is not safe for an external script to do), then a message to this effect needs to be provided so the user understands that one more step needs to be done manually. - Michel.

When executing the MtkInstaller-1.5.0.pkg package, the progress bar in the window that opens while the process is on-going moves quickly initially, and then remains stuck for between 15 and 20 min (on my computer), perhaps due to the time required to download external libraries through a slow Internet connection. If that progress bar cannot be made to move (even slowly) while this happens, it would be useful to include a message indicating that this process can take that much time, to avoid users killing the process out of impatience. - Michel.

The Mtk function MtkFindFileList does not work when trying to locate AGP files: the names of those files do not feature an orbit number, and the C code does not foresee the possibility of not specifying an orbit, as far as I can see... Here is what happens in IDL:

First, make sure the file exists:

IDL> res = FILE_SEARCH("/Users/michel/Codes/Julia/JMtk15/data/MISR/*_AGP_*")
IDL> res
/Users/michel/Codes/Julia/JMtk15/data/MISR/MISR_AM1_AGP_P168_F01_24.hdf

Then try to find it:

IDL> search_dir = "/Users/michel/Codes/Julia/JMtk15/data/MISR/"
IDL> product = "AGP"
IDL> path = "168"
IDL> version = "F01_24"
IDL> status = MTK_FIND_FILELIST(search_dir, product, camera, path, orbit, version, cnt, files)
% MTK_FIND_FILELIST: String expression required in this context: CAMERA.
% Execution halted at: $MAIN$          
IDL> camera = ""
IDL> status = MTK_FIND_FILELIST(search_dir, product, camera, path, orbit, version, cnt, files)
% MTK_FIND_FILELIST: String expression required in this context: ORBIT.
% Execution halted at: $MAIN$          
IDL> orbit = ""
IDL> status = MTK_FIND_FILELIST(search_dir, product, camera, path, orbit, version, cnt, files)
IDL> files
% Attempt to call undefined procedure: 'FILES'.
% Execution halted at: $MAIN$          
IDL> PRINT, files
% PRINT: Variable is undefined: FILES.
% Execution halted at: $MAIN$          

So the function requires all inputs to be provided explicitly, but does not locate the desired file because the latter does not include an orbit number. The code should probably be modified in such a way that whenever the product type is AGP, whatever value is given to camera and orbit is ignored.

The same problem occurs when calling the C function directly from within Julia, so this is not an IDL-specific problem.

Failures in the install_hdf+hdfeos script are easily missed because the script does not stop on errors.

Recommend adding 'set -e' at the top of the script to automatically exit on errors.

Here is another slight inconsistency in Mtk: When calling MtkFileType to request the file type of either GRP ELLIPSOID or TERRAIN files, the answer includes the "_GM" (Global Mode) identifier, but when doing the same for an RCCM file, that suffix is absent, even though it is part of the file name. I know there is no Local Mode version of the RCCM file, but when reconstructing a file name from its parts, there must be additional logic to add this suffix when an RCCM file name is generated.

For instance, function MtkFindFileList to find RCCM files and function MtkMakeFilename to construct the name of a RCCM file both require to specify the file type as GRP_RCCM_GM, not as GRP_RCCM, and yield incorrect results otherwise.

Python built against nurses 6 segfaults at misr_init.c:56, assigning nline to the global nl.

python -c "import MisrToolkit as mtk; mtk.MtkRegion(50,100,100)"

Digging around in GDB, the symbol "nl" is defined twice:

(gdb) info address nl
Symbol "nl" is at 0x2aaaaacee6b0 in a file compiled without debugging.

(gdb) info function nl
All functions matching regular expression "nl":
...
0x00002aaaaacee6b0  nl
...

(gdb) info variable nl
All variables matching regular expression "nl":
...
0x00002aaaad0ef1d4  nl
...

The segfault is the result of trying to write to the address of a function. This can be confirmed by LD_PRELOADing MisrToolkit.so. The dynamic linker chooses the variable definition first and the segfault is avoided (although if anything calls nl the function, things will, of course, segfault). The function nl is defined in ncurses 6:

nm -D libncurses.so.6.0 | grep nl
00000000000216b0 T nl

Changing nl to a different name fixes the segfault entirely, which I have tested.

For clarity and completeness, copy the README, CHANGELOG and LICENSE files to the appropriate /Applications/Mtk-1.5.0/ subdirectory when the Mtk is installed on a Mac using the current MtkInstaller-1.5.0.pkg package: currently, those files are only 'visible' when downloading the source code. - Michel.

The following URLs in scripts/download_libraries are no longer valid:

curl -L -O https://observer.gsfc.nasa.gov/ftp/edhs/hdfeos/previous_releases/zlib-1.2.5.tar.gz
curl -L -O https://observer.gsfc.nasa.gov/ftp/edhs/hdfeos/previous_releases/jpegsrc.v6b.tar.gz
curl -L -O https://observer.gsfc.nasa.gov/ftp/edhs/hdfeos/previous_releases/HDF-EOS2.18v1.00.tar.Z

HDF-EOS libraries are moved to the Earthdata Wiki:
https://wiki.earthdata.nasa.gov/display/DAS/Toolkit+Downloads

It is unclear how to get direct links from the Earthdata Wiki that work with curl.

In the IDL documentation of function MTK_RAD_TO_DD (Mtk-1.5.0/MISR-Toolkit-1.5.0/doc/IDL_HTML_DOCS/mtk_rad_to_dd.html), the arguments in the first instance of the call to this function (in the I/O section) have been inverted, though they are correct in the subsequent Examples section.

Replace MTK_RAD_TO_DD( dd, rad ) by MTK_RAD_TO_DD( rad, dd ).

The IDL documentation for function MTK_RAD_TO_DD states that

IDL> radians = 1.1440634
IDL> status = MTK_RAD_TO_DD(radians, decdeg)
IDL> PRINT, decdeg
       65.550002
IDL> PRINT, decdeg, FORMAT = '(D18.12)'
   65.550001653340

where the second PRINT statement prints the same result with more decimals. Yet, calling the corresponding C function MtkRadToDd from Julia with the same input yields

julia> decdeg = jMtkRadToDd_tst_1()
65.55000431538731

These two results do not differ by much, but I would not have expected such a difference given that both computations are presumably conducted in double precision (double or Float64). It may be worth checking that the IDL version of this function either calls the C function with the correct argument or implements the computation in double precision.

HDFView 3.1.4 reports that field "Leaf_Area_Index_Best_Estimate" of the file "MISR_AM1_AS_LAND_P168_O068050_F08_0023.nc" has the following 6 attributes:

DIMENSION_LIST
_FillValue
coordinates
long_name
standard_name
units

However, the Mtk function 'MtkFieldAttrList' (called from Julia) reports only 5 attributes, in a different order:

julia> attrlist
5-element Vector{String}:
 "_FillValue"
 "coordinates"
 "units"
 "standard_name"
 "long_name"

I have never noticed that Julia modifies the order of objects returned by C, so I suspect that the C function itself returns those attributes in that order. This raises two questions:

4.1. Why is HDFView reporting more attributes than Mtk? Are some of those attributes "hidden" for certain purposes?

4.2. Why is the order of the attributes different in the HDFView and C lists? The order may not matter if one looks for an attribute by name, but it would lead to errors if one relied on the order reported by HDFView to retrieve an attribute value with the C functions on the basis of its rank in the list.

There is a slight inconsistency in the generation of AGP files (version F01_24): All file attributes are continuous strings (without blank spaces inserted), except for the penultimate item, which is set to "End Block" instead of the expected "End_Block":

IDL> filename = "/Users/michel/MISR_HR/Input/AGP/MISR_AM1_AGP_P005_F01_24.hdf"
IDL> status = MTK_FILEATTR_LIST(filename, attrcnt, attrlist)
IDL> attrlist[25:26]
Start_block
End block

This appears to be the case for all 180 AGP files. If those are going to be updated as part of the re-processing, it might be opportune to correct this and enhance the internal consistency of those file attributes.

Mtk function MtkFileBlockMetaList does not work with MISR AS_LAND (and probably AEROSOLS) files, because it always assumes the files to be in HDF format:

IDL> filename = root + 'MISR_AM1_AS_LAND_P168_O068050_F08_0023.nc'
IDL> status = MTK_FILE_BLOCK_META_LIST(filename, nblockmeta, blockmetalist)
% MTK_FILE_BLOCK_META_LIST: HDF HDFOPEN FAILED

This error also arises when calling the C function from Julia. If such NetCDF files do not have block metadata information, I would have expected the function to return quietly, or perhaps to issue a warning, but not to cause an error condition that stops IDL or Julia processing. And if they do include such metadata, the function should be updated to handle those files.

The MtkInstaller-1.5.0.pkg package apparently looks for the IDL executable in the folder where that software product is installed when the user buys a license for that software alone. However, if one acquires (currently from Harris) the combined ENVI/IDL package, then the IDL executable is located in a subdirectory of the ENVI installation. The Mtk installation script does not find IDL and does not generate the binary files 'idl_MisrToolkit.dlm' and 'idl_MisrToolkit.so'. The suggestion is (1) if possible, to modify the script to look in at least those two places for the IDL executable (or search for it dynamically), and (2) to issue an error message if IDL is not found, so that the user knows that the installation is not complete, although the script closes by announcing successful completion. - Michel.

When applied to a GP_GMP data file (specifically MISR_AM1_GP_GMP_P168_O068050_F03_0013.hdf), function MTK_FILE_COREMETADATA_QUERY causes IDL to crash and quit, while calling the corresponding C function MtkFileCoreMetaDataQuery from Julia (on the same file) causes a segmentation fault!

The function does work correctly with other HDF files (I have tested it with AGP, RCCM, ELLIPSOID, TERRESTRIAL, and TC_CLOUD), as well as with the AS_LAND NetCDF file. If this function is not supposed to return useful information for GP_GMP files, I would expect it to check the nature of the input file and return a warning or an error message, rather than crashing...

A similar discrepancy occurs with this function too:

IDL> radians = 1.14406333
IDL> status = MTK_RAD_TO_DEG_MIN_SEC(radians, degrees, minutes, seconds)
IDL> PRINT, degrees, minutes, seconds
          65          33    0.0059520230

while calling the C function directly from Julia, with the same inut yields

julia> degrees, minutes, seconds = jMtkRadToDegMinSec_tst_1()
(65, 33, 0.0010968595743179321)

Hence, same suggestion: Check that the IDL version of this function either calls the C function with the correct (un-truncated) argument or implements the computation in double precision.

Function MtkMakeFilename aims to generate the standard filename of a MISR product, given the necessary inputs (specifically, directory, product, camera, path, orbit, version). This works fine for HDF files, but not for NetCDF files: the filename is actually correct, but the file extension is systematically set to .hdf instead of .nc. This used to be correct, but AS_LAND (and AEROSOLS) files have been formatted as NetCDF files for quite some time now.

This function should be updated to handle both the older and the newer formats, based on the file version identifier.