dawhite / mctk Goto Github PK

Dear Prof.:
I am a student. I appreciate that the MCTK helped me make many things. However, a error occurred when I process a MODIS L1B file, today. I am not able to fix this error. After checking processor, i think the error was trigered by the output polar, projection, which is my needness. Please give me some advices. thanks!!!

the code:

pro test_batch_modis_conversion_level_1b
  compile_opt idl2
  E = ENVI(/HEADLESS)
  modis_l1b_file = 'J:\old_modis\20151222_2\MOD021KM.A2015356.1640.061.2017323113709.hdf'
  ;The specified output location MUST end in the appropriate path
  ;separator for your OS
  output_location = 'J:\old_modis'
  output_rootname = 'level_1b'
  ;Calibration method schema is:
  ;0 = Radiance \ Emissivity, 1 = Reflectance \ Emissivity
  calib_method = 1
  ;Output method schema is:
  ;0 = Standard, 1 = Projected, 2 = Standard and Projected
  out_method = 1
  output_projection = envi_proj_create(/geographic,PE_COORD_SYS_CODE=102021); 102021 is a code of projection stored in EnviPEProjcsStrings.txt 
  ;Use bow tie correction during the projection process
  convert_modis_data, in_file=modis_l1b_file, out_path=output_location, $
    out_root=output_rootname, /l1b, out_method=out_method, $
    out_proj=output_projection, calib_method=calib_method, /bowtie, $
    sd_pos=[0], /no_msg, background=0.0
end

the result: (the background image is Landsat 8 OLI data, and error MODIS located upper left)

Hi dawhite,

I'm currently trying to install the MCTK plugin into ENVI but have hit a strange wall.

I'm placing the two files (mctk.sav & modis_products.scsv) into the directories advised in the user guide.. When I open ENVI Classic, I can go into File>OpenExternalFile>EOS>MODIS Conversion Toolkit and it all opens as expected.

However when I open the normal ENVI, I can see MCTK in the extensions on the right hand side, but when I double click it, i get the error "Modis_Products.scsv could not be found".

The files are all in the same places i.e \extensions and \save_add and i've ensured security rights to the files are ok, so I'm abit stumped and hoping you can help.

Thanks,

Rich

Hi dawhite,
When I used MCTK under IDL 8.5 to geolocate MOD09 files, at the step "Performing final grid operation" , MCTK was stuck and had no response. The timeout of thread of PC was also happened. I think the compatibility of MCTK may be the reason. Would you please check this?
Cheers,
JZH

Hi dawhite,
Your MCTK is a useful tool for processing MODIS products. Thank you very much.
When I use MCTK to process MOD04_3K product under ENVI/IDL 4.5 on Windows 7, the interactive way works properly; however, the programming way doesn't produce any outputs. The following is my script, which is customized based on the Level 2 swath example from the user's guide.

; Level 2 swath example
PRO TEST_BATCH_MODIS_CONVERSION_L2_SWATH
  COMPILE_OPT idl2

  modis_swath_file = 'C:\MCTK_Input\MOD04_3K.A2014335.0310.006.2015030045227.hdf'

  ; The specified output location MUST end in the appropriate path
  ; separator for your OS
  output_location = 'C:\MCTK_Output\'

  output_rootname = 'AOD'
  swath_name = 'mod04'

  sd_names = ['Optical_Depth_Land_And_Ocean', 'Aerosol_Type_Land']

  ; Output method schema is:
  ; 0 = Standard, 1 = Projected, 2 = Standard and Projected
  out_method = 1

  output_projection = ENVI_PROJ_CREATE(/geographic)

  ; Choosing nearest neighbor interpolation
  interpolation_method = 0

  ; do not put the bridge creation/destruction code inside a loop
  bridges = mctk_create_bridges()

  convert_modis_data, in_file=modis_swath_file, $
    out_path=output_location, out_root=output_rootname, $
    swt_name=swath_name,$
    sd_names=sd_names, $
    out_method=out_method, out_proj=output_projection, $
    interp_method=interpolation_method, /no_msg, $
    r_fid_array=r_fid_array, r_fname_array=r_fname_array, $
    bridges=bridges,msg=msg

  mctk_destroy_bridges, bridges
END

The same problem also occurs under ENVI/IDL 5.3. Moreover, under ENVI/IDL 5.3, the Level 2 swath example can't work and doesn't produce any outputs.

Hi,
I am new with this plugin and I found that it does almost all of thing that I need.But I want to know does this plugin performs an atmospheric correction before converting DN values to brightness temperature?Or I need perform it myself using FLAASH?
Is there any problem if I run FLAASH after calculating BT using this plugin?

Thank you very much and sorry if this question is not related to any bug or things like that
Thanks
Majid

I have .pro cod which I want them not to be modified by others. how could i convert .sav into read only files?

I do the Steps include follow as:

1. Reset your session
   IDL>.FULL_RESET_SESSION or
   IDL>.F
2. Open the ENVI procedure (*.pro)
   IDL> .COMPILE my_envi_function.pro
3. Call RESOLVE_ALL
   IDL> Resolve_All, /continue_on_error, skip_routines='envi'
4. Create the ENVI save file (*.sav)
   IDL> SAVE, file='my_envi_function.sav', /routines

but when deleted .pro file then, .sav file is deactivate!!!!??

Dear dawhite:
I'm sorry to bother you again. I have a new problem when I try to batch re-project mosaiced HDF data. The mosaiced HDF data including 'h27v05' 'h28v05' 'h28v06' 'h29v06'. And I can't running successfully. The IDL and ENVI just unreacted. Consequently, could you tell me what to do? There are my codes , mosaiced HDF data's map info (which is mosaiced MOD11A2_2000065.HDF) and unreacted information. Thanks!
Cheers,
ZB

Hi Dear all
I'm going to georeference several MODIS image(300) and product(300).I can not find some function in MCTK such as "convert_modis_data","mctk_create_bridges" and ' mctk_destroy_bridges''and etc. Where can I download this file?

For some MODIS L1b images, i get the following error from ENVI while performing the bowtie correction:

TRIANGULATE: Not enough valid and unique points specified.

example of product: MOD02HKM.A2013286.1450.005.2013295174309.hdf

do you have any idea where it can originate from?

If you need more processing parameter i can put my script in a gist.

Regards

Hi, I am trying to use MCTK (v 2.1.4) in ENVI 5.1 to georeference MODIS Level 2 swath snow product (MYD10_L2). However, I am getting an error 'Tag name BOWTIE is undefined for structure'. I was wondering why I am getting this error? I would greatly appreciate your advice. Thank you.

Hi, I have processed the MODIS L1B data via the MCTK toolkit. The original data has been processed into the Reflectance and Emissive data.
But I found the maxmium value of some spectral bands is higher than 1.0.
Why is it happen? Shouldn't the data is in the range of 0 to 1.0?

Dear Dawhite：
When I use the MCTK-2.1.11, the error is about "keyword GRID not allowed in call to: CONVERT_MODIS_DATA". I read your comments about “Jennyzht” on 16 Dec 2017(“https://github.com/dawhite/MCTK/issues/7”). And I delete older version of the MCTK, But the error still exists. Besides. I read some answers, which about path conflict and path priority. Whereas, I didn’t do it. Can you help me, and give me some advice?
Cheers,
ZB
code.docx

Hi,
I am using idl to loop for about 400 hdf files..But I faced this error using mckt,Can you please help me to find the rresion of this error?

Thank you
`<
;envi

print,'Read hdf file';
LIST=FINDFILE('D:\hdf*.hdf');
bridges = mctk_create_bridges()

FOR f=0,(LIST.LENGTH-1) Do begin
print,'read file '+ string(f) + ' of '+string(LIST.LENGTH)

print,'read file '+ string(f) + ' of '+string(LIST.LENGTH)
modis_swath_file = LIST[f]
;The specified output location MUST end in the appropriate path
;separator for your OS
output_location = 'D: \MCTK_Output'
output_rootname = 'level_2a'
;Output method schema is:
;0 = Standard, 1 = Projected, 2 = Standard and Projected
out_method = 2
output_projection = envi_proj_create(/geographic)
;Choosing linear interpolation
interpolation_method = 1
;do not put the bridge creation/destruction code inside a loop
sd_names = [ 'LST_Day_1km' , 'LST_Night_1km' ]
convert_modis_data, in_file=modis_swath_file, $
out_path=output_location, out_root=output_rootname, $
sd_names=sd_names, $
out_method=out_method, $
interp_method=interpolation_method, $
bridges=bridges, msg=msg
ENDFOR
mctk_destroy_bridges, bridges

print,msg
end>`

Dear dawhite,
When I run the example code from mctk user guide, I found that it's not success. The msg showed that Unable to locate the specified swath. Check name against HDF file contents. I want to know how to solve this problem. Thanks!

;Level 2 swath example
pro test_batch_modis_conversion_l2_swath
compile_opt idl2

;modis_swath_file = 'C:\MCTK_Input\MOD07_L2.A2013173.0710.006.2013173195131.hdf'

modis_swath_file = 'D:\TestData\MYD04\MYD04_L2.A2019001.0250.061.2019001165255.hdf'
;The specified output location MUST end in the appropriate path
;separator for your OS
output_location = 'D:\Data\AOD\MODIS_10KM\TestResult'

output_rootname = 'MYD04_A2019001'

swath_name = 'MYD04'

sd_names = ['Optical_Depth_Land_And_Ocean']

;Output method schema is:
;0 = Standard, 1 = Projected, 2 = Standard and Projected
out_method = 1

output_projection = envi_proj_create(/geographic)

;Choosing nearest neighbor interpolation
interpolation_method = 0

;do not put the bridge creation/destruction code inside a loop
bridges = mctk_create_bridges()

convert_modis_data, in_file=modis_swath_file, $
out_path=output_location, out_root=output_rootname, $
swt_name=swath_name, sd_names=sd_names, $
out_method=out_method, out_proj=output_projection, $
interp_method=interpolation_method, /no_msg, $
r_fid_array=r_fid_array, r_fname_array=r_fname_array, $
bridges=bridges, msg=msg
print, msg
mctk_destroy_bridges, bridges

end

Thank you for your plugin to process MODIS data. However, after my processing of MODIS L1B data by MCTK, I found the value of some pixels are NaN, I wonder to konw the reasons. thank you!

Error occurs when I open it as follows:

A serious error happened when I used MCTK to process MOD03 data, like MOD03.A2014001.0615.006.2014001134042.hdf.

Hello Devin,

Thank you for your great toolkit! I am trying to use it to convert several MODIS products (MOD04_L2, MOD04_3K, MOD05_L2). I am an intermediate level programmer, but new to the IDL language, so the problem I've come across may not be specific to your creation, but rather an IDL problem.

I wrote a batch script to process ~40 MOD04 and MYD04 files, based off of your template but with an ugly loop that iterates for each HDF files. It ran smoothly until certain HDF files couldn't be processed, yielding an error in the IDL console:

Unable to attach to supplied HDF file using swath approach

When I try to process these same files using the MCTK GUI, an error occurs when picking the input HDF file:

Warning: "Expression must be a structure in the context: MODIS_INFO"

This error occurs for all files beginning with MOD, so MOD04 as well as 05 and 07. Strangely my code as it currently stands works for all MYD files, so it seems at least. I had originally written my script for files with a MYD filename prefix, then modified the string variables for the MOD files, but otherwise it functions the same way.

My batch script works for MYD files, but strangely, certain MYD files selected from the MCTK GUI yield the same "Expression must be a structure in the context: MODIS_INFO" error.

I've been scratching my head to figure out why my code seemed to work perfectly fine before, but is now failing for 'random' files, both in batch mode and with the GUI! I hope you have some advice for me in these frustrating times...

Kind regards,

Jigme

P.S.
The filename for a sample HDF file that fails is "MOD04_L2.A2005251.0330.006.2014349063845.hdf" if that clarifies anything.

-------------------------------Draft code below -------------------------------

PRO whole_batch_script_MOD
  
COMPILE_OPT IDL2
  
;envi, log_File = 'batch.txt' ;, /restore_base_save_files
;********************* SETTINGS *****************************
;************************************************************
; --- Directory Settings --- 
input_dir = 'C:\MCTK_Input - Copy\'
output_location = 'C:\MCTK_Output\'

; --- Projection Settings ---
out_proj   = envi_proj_create(/utm, zone=47, datum='WGS-84')
interp_method = 1  ; Bilinear
out_method  = 1 ; 1 for Projected, 2 for projected and standard

; --- Data subset selection ---
sd_selection_mod04_3k = ['Optical_Depth_Land_And_Ocean','Image_Optical_Depth_Land_And_Ocean','Aerosol_Type_Land','Corrected_Optical_Depth_Land','Quality_Assurance_Land','Deep_Blue_Aerosol_Optical_Depth_550_Land','Deep_Blue_Aerosol_Optical_Depth_Land','Cloud_Mask_QA']
sd_selection_mod04    = ['Optical_Depth_Land_And_Ocean','Image_Optical_Depth_Land_And_Ocean','Aerosol_Type_Land','Corrected_Optical_Depth_Land','Quality_Assurance_Land','Deep_Blue_Aerosol_Optical_Depth_550_Land','Deep_Blue_Aerosol_Optical_Depth_Land','Cloud_Mask_QA']
sd_selection_mod05    = ['Cloud_Mask_QA','Water_Vapor_Near_Infrared','Quality_Assurance_Near_Infrared']
sd_selection_mod07    = ['Cloud_Mask','Surface_Pressure','Surface_Elevation','Processing_Flag','Total_Ozone','Water_Vapor','Quality_Assurance_Infrared']
;************************************************************
sub_directory = file_search(input_dir,'*.hdf',COUNT=filecount)
print,'Filecount is:',filecount


counter = 1 ; Counter to make list of Geolocation files
geoloc_id = MAKE_ARRAY(filecount,1, /string);
geoloc_dir = MAKE_ARRAY(filecount,1, /string);

print,'--------------- Searching for and tagging Geolocation files ---------------'
  FOR i=0,filecount-1 DO BEGIN
    ;print,'Iteration number:',i
  ;************************************************************
    subdir        = sub_directory[i] ;'_b1.tif'  ;?? jumpl is a label for the lines following it?
    subdir_name   = file_basename(subdir)
    prod_name      = STRMID(subdir_name,0,5)
 
    ;************************************************************
    IF (prod_name EQ 'MOD03') or (prod_name EQ 'MOD03') THEN BEGIN
    print, STRMID(subdir_name,0,20),'... is a geolocation file'
    geoloc_dir[counter] = subdir
    geoloc_id[counter] = STRMID(subdir_name,6,17)  ; Capture date and time as ID
    counter = counter+1
    ;print,subdir_name,geoloc_id
    ;************************************************************
    ENDIF ELSE CONTINUE 
  ENDFOR 
  
;bridges = mctk_create_bridges()

 counter = 1
 FOR i=0,filecount-1 DO BEGIN
   ;************************************************************
    subdir        = sub_directory[i] ;
    subdir_name   = file_basename(subdir)
    prod_name     = STRMID(subdir_name,0,7)
        ;************************************************************
    IF (prod_name EQ 'MOD03.A') or (prod_name EQ 'MOD03') THEN BEGIN
      print,'File is geolocation file',prod_name,' | Skipping....' 
      CONTINUE
    ENDIF 
    ;************************************************************
    print,'--------------- Matching MOD03 file to Swath file:',counter,'  ---------------'
    print,subdir
    swath_id = STRMID(subdir_name,9,17)
    geoloc_index = WHERE(geoloc_id EQ swath_id, geoloc_filecount) ; Count numer of elements found
    IF geoloc_filecount EQ 1 THEN print,'Geolocation file identified as: ',geoloc_dir[geoloc_index] $; Only perform action if SINGLE match is found
    ELSE BEGIN 
      print,'ERROR - No file or multiple matching files found ERROR',subdir_name
      BREAK
    ENDELSE 
    ;******************** Get Root Name**************************
    root_prod = STRMID(prod_name,0,5)
    root_date = STRMID(subdir_name,10,7)
    root_time = STRMID(subdir_name,18,4)
    output_rootname = root_prod +'_' +root_date +'_' + root_time +'_' + 'UTM47'
    ;************************************************************
    IF (prod_name EQ 'MOD04_3') THEN BEGIN
      print,'--Product identified as MOD04_3K--'
      swath_name = 'mod04'
      sd_names = sd_selection_mod04_3k
    ENDIF ELSE $
    IF (prod_name EQ 'MOD04_L') THEN BEGIN
      print,'--Product identified as MOD04_L2--'
      swath_name = 'mod04'
      sd_names = sd_selection_mod04
    ENDIF ELSE $
    IF (prod_name EQ 'MOD05_L') THEN BEGIN
      print,'--Product identified as MOD05_L2--'
      swath_name = 'mod05'
      sd_names = sd_selection_mod05
    ENDIF ELSE $
    IF (prod_name EQ 'MOD07_L') THEN BEGIN
      print,'--Product identified as MOD07_L2--'
      swath_name = 'mod07'
      sd_names = sd_selection_mod07
    ENDIF ELSE BEGIN
    print,'ERROR ERROR: Product not matched with any identifiers. ERROR ERROR File name iteration of failure:'
    print,subdir_name,'....',i
    ENDELSE
    ;************************************************************
    modis_swath_file = subdir
    
    geoloc_file = (string(geoloc_dir[geoloc_index]))[0]    ; Assign geolocation file, found by matching Geoloc ID
    
    convert_modis_data, in_file=modis_swath_file, $
      out_path=output_location, out_root=output_rootname, $
      swt_name=swath_name, sd_names=sd_names, geoloc_file = geoloc_file, $
      out_method=out_method, out_proj=out_proj, $
      interp_method=interp_method, /use_double, /no_msg, /progress, $
      r_fid_array=r_fid_array, r_fname_array=r_fname_array, $
      msg = msg;bridges=bridges, msg = msg

    print,'MCTK done :)!'
    
    if msg ne '' then print, msg


    ;************************************************************   
    counter = counter + 1
  ENDFOR

;mctk_destroy_bridges, bridges

print,'----------- BATCH PROCESSING COMPLETE, HAVE A NICE DAY -----------'

  ;CATCH, Error_status ; Error_status returns the index of the error
  ;; This statement begi ns the error handler:
  ;  IF Error_status NE 0 THEN BEGIN
  ;    STR_ERROR = 'An error has occured while processing the file:   '+file_basename(sub_directory[i])+'!'
  ;    PRINT, STR_ERROR
  ;    Error_status=0
  ;    strlin='the file:'+FILE_basename(sub_directory[i])+' has failed!'
  ;    printf,lun, strlin
  ;    i=i+1
  ;    goto, jump1
  ;  ENDIF
  ;

END

Hi dawhite,
Can you give some APIs to support custom georegistration? For example, I can get the geolocated data by your APIs using latitude and longitude bands and my retrieved data band from MODIS L1B and MOD03 data.
Cheers!

Hello, dawhite.
The function mctk_create_bridges use all CPU in my computer in the default option. Dose it have some parameters that can control the number of CUP used?
Thanks！

Dear dawhite

When i run idl code like this,

;Level 1B example
pro test_batch_modis_conversion_level_1b
compile_opt idl2

modis_l1b_file = 'D:\OmegaDrive\Thesis\SARA algorithm\011117\MOD02HKM.A2017305.0355.061.2017305175655.hdf'

;The specified output location MUST end in the appropriate path
;separator for your OS
output_location = 'E:\SARA'

output_rootname = 'TOA'

;Calibration method schema is:
;0 = Radiance / Emissivity, 1 = Reflectance / Emissivity, 2 = Radiance / Brightness Temp
calib_method = 2

;Output method schema is:
;0 = Standard, 1 = Projected, 2 = Standard and Projected
out_method = 1

output_projection = envi_proj_create(/geographic)

;do not put the bridge creation/destruction code inside a loop
bridges = mctk_create_bridges()

;Choosing linear interpolation
interpolation_method = 1

convert_modis_data, in_file=modis_l1b_file, out_path=output_location, $
out_root=output_rootname, out_method=out_method, $
interp_method=interpolation_method, $
out_proj=output_projection, calib_method=calib_method, $
sd_pos=[1,3], /no_msg, background=0.0, r_fid_array=r_fid_array, $
r_fname_array=r_fname_array, bridges=bridges, msg=msg

mctk_destroy_bridges, bridges

end

the result from this code does not appear, it provides data (.dat) like this photo. How to solve?

I am trying to convert a hdf file to another projection in IDL , the command following is what i write:

`;Level 1A example
compile_opt idl2
PRINT, 'START : ',SYSTIME()
MODIS_FILE = 'f:\guizhou\data\modis\MYD05_L2.A2016041.0600.006.2016041204845.hdf'
OUTPUT_LOCATION = 'f:\guizhou\data\modis'
output_rootname = 'wv_ni'
OUTPUT_METHOD = 1 ;0 = Standard, 1 = Projected, 2 = Standard and Projected

GRID_NAME = "Water_Vaper_Near_Infrared"
;投影转换设定
;差值设置
INTERPOLATION_METHOD = 0 ;0 — Nearest neighbor

CONVERT_MODIS_DATA, IN_FILE=MODIS_FILE, OUT_PATH=OUTPUT_LOCATION, OUT_ROOT=output_rootname $
,GD_NAME = GRID_NAME $
,OUT_METHOD=OUTPUT_METHOD $
,INTERP_METHOD=INTERPOLATION_METHOD $
;,BACKGROUND='0', FILL_REPLACE_VALUE='0' $
;/no_msg,$
,R_FID_ARRAY=R_FID_ARRAY, R_FNAME_ARRAY=R_FNAME_ARRAY, /NO_MSG
END`

when I run it, it says

% Compiled module: $MAIN$.
START : Sun Dec 17 13:13:02 2017
% Attempt to call undefined procedure/function: 'ENVI_QUERY_VERSION'.
% Execution halted at: GET_SAVE_ADD_PATH
% CONVERT_MODIS_DATA
% $MAIN$ 14 C:\Users\cyeme\IDLWorkspace83\Guizhou\modis_covert_wv.pro

why is that ? I use ENVI 5.1

Hello Devin,
Thank you for your excellent toolkit. I extracted and reprojected AOD images from the MOD04_3K dataset using the MCTK 2.17. To my knowledge, the valid values of AOD are usually spatially continuous, so are the bad or fill value. But I found that many fill values appear within the good part of my resulting AOD images, so as to form a fishnet. This is not good for my study. And this does not occur in previous versions, but MOD04_3K is not applicable.
As a newcomer of programming, could you please tell me the reasons or give me any suggestions?
I really appreciate your help.

Boorn

I have installed ENVI 5.0 and want to process the Modis aerosol data. I put both files in the corresponding folders:
C:\Program Files\Exelis\ENVI50\extensions
C:\Program Files\Exelis\ENVI50\classic\save_add
but the toolkit extension is not appearing in both modes classic and standard ? can someone help me to resolve the issue ?

Thanks

Can cmtk batch read and bulk process hdf?

The georeferenced Height data in MOD03 show wrong result as an attached figure.

This following is the used code:
PRO geolocate_MOD03
COMPILE_OPT idl2

FILE='E:\tmp\MOD03.A2008124.0415.006.2012243171834.hdf'
output_location='E:'
rootname = 'M03'
swath_name='MODIS_Swath_Type_GEO'
sd_names='Height'

;Output method schema is:
;0 = Standard, 1 = Projected, 2 = Standard and Projected
out_method = 1

output_proj = ENVI_PROJ_CREATE(/geographic)

;the resampling method
;0 — Nearest neighbor
;1 — Linear
;2 — Cubic convolution
interp_method = 0

backvalue=0.0

convert_modis_data, in_file=file, $
out_path=output_location,out_root=rootname,$
out_method=out_method,interp_method=interp_method,$
out_proj=output_proj,background=backvalue,$
swt_name=swath_name, sd_names=sd_names, $
r_fname_array=r_fnames,msg=msg
END

Hi, I'm using the IDL + ENVI to process the MOD12A3 data. The MCTK GUI works very well, but the function "convert_modis_data" never become activated in IDL. I have the mctv.sav and modis_products.scsv in both save_add and extensions folders. It seems someone reported the similar problems online. But I haven't find out the solution.

Would you please provide a advise?

Thanks!