Level-2A processor used for atmospheric correction and cloud-detection. The active repository is the one below, this one is kept to leave access to the older issues.
Home Page: https://gitlab.orfeo-toolbox.org/maja/maja
License: Apache License 2.0
Hi!
I am trying to run MAJA (v.3.3.2) without CAMS option on L1C tiles downloaded from peps.
At the moment I am getting such error:
vns::Data::ERROR: EarthExplorerXMLFileHandler(0x7af7d0): XMLFileHandler::LoadFile: Error while loading the file [.../Start-MAJA/temp_files/T22KHB/22KHB/GIPP_united/in/S2A_TEST_GIP_L2COMM_L_ALLSITES_00001_20190626_21000101.EEF]. Parsed with errors.
Will be grateful for any ideas or suggestions.
In the readme you mention that you can get the parameters folder in a version using CAMS and one not using CAMS and that the one using CAMS is rather big. When you click the link to the one NOT using CAMS: http://tully.ups-tlse.fr/olivier/gipp_maja/tree/master/GIPP_MAJA_3.1.2_TM and reads the readme, you are told that "For data volume reasons, the directory is not complete, and you should add the files contained in this reference". When you follow that link you end up at: https://zenodo.org/record/2553164 which seems to be the LUTs for the version WITH CAMS? It seems to be the same (although an older version) LUTs you get when picking configuration for the version actually using CAMS. When you unpack the data you get from https://zenodo.org/record/2553164 you get the GIPP_MAJA_3.1.2_TM_CAMS folder. It's not clear (1) if I can use that for the version running without CAMS and (2) if this is the complete configuration dir or I have to copy its content into my configuration folder which I got from: http://tully.ups-tlse.fr/olivier/gipp_maja/tree/master/GIPP_MAJA_3.1.2_TM?
I ran start_maja.py for a while and it seems like the first date seems to be processed successfully (I got the processed TIFs for the first date), but then I got an error before starting on the second image: 2019-05-09 18:46:11,809 - Start-Maja - ERROR - First backward processing was unsuccessful, check MAJA installation .
When I check the log in the ouput directory I did find a GDAL error occuring several times at the end of the file, but I am not sure if this is the cause of the exit:
2019-05-10T09:18:51.621064 ip-172-31-2-35 maja-processing-3.2.2 3.2 [000000024200] [D] => Caching with file name vns_caching_TOAReader5.tif run in 2.8 minutes. [vnsSentinel2L1ImageFileReaderBase.txx:GenerateTOACaching:490]
2019-05-10T09:18:51.723171 ip-172-31-2-35 maja-processing-3.2.2 3.2 [000000024200] [D] Caching the </home/ubuntu/efs/Start-MAJA/tmp/site_name/T33VWC/GIPP_001/in/S2B_MSIL1C_20181024T102059_N0206_R065_T33VWC_20181024T160131.SAFE/GRANULE/L1C_T33VWC_A008528_20181024T102428/IMG_DATA/T33VWC_20181024T102059_B05.jp2> image filename... [vnsSentinel2L1ImageFileReaderBase.txx:GenerateTOACaching:477]
2019-05-10T09:18:51.723212 ip-172-31-2-35 maja-processing-3.2.2 3.2 [000000024200] [D] Reflectance quantification value : 0.0001 [vnsSentinel2L1ImageFileReaderBase.txx:GenerateTOACaching:478]
2019-05-10T09:18:51.766922 ip-172-31-2-35 maja-processing-3.2.2 3.2 [000000024200] [D] GDAL Error 1 : /home/ubuntu/efs/Start-MAJA/tmp/site/T33VWC/GIPP_001/in/S2B_MSIL1C_20181024T102059_N0206_R065_T33VWC_20181024T160131.SAFE/GRANULE/L1C_T33VWC_A008528_20181024T102428/IMG_DATA/T33VWC_20181024T102059_B05.jp2:Not a TIFF or MDI file, bad magic number 0 (0x0) [vnsGDALLogInit.cxx:CPLIPFErrorHandler:59]
2019-05-10T09:18:51.797647 ip-172-31-2-35 maja-processing-3.2.2 3.2 [000000024200] [D] vnsCachingMacro Proceed caching... with file name vns_caching_TOAReader6.tif (ModeStreamDivisions=5; NumberOfStreamDivisions=800). [vnsSentinel2L1ImageFileReaderBase.txx:GenerateTOACaching:490]
I have also tried looking into the code of start_maja.py and it seems that at some point no files are found for nomL2init_Natif and nomL2init_MUSCATE line 396-402), resulting in L2type to be set to None which does result in an error later (line 491-495) (I don't see any other place where L2type is defined in between).
nomL2init_Natif = glob.glob("%s/%s" % (repL2, nomL2_par_dateImg_Natif[d]))
nomL2init_MUSCATE = glob.glob("%s/%s" % (repL2, nomL2_par_dateImg_MUSCATE[d]))
if len(nomL2init_Natif) > 0:
derniereDate = d
L2type = "Natif"
elif len(nomL2init_MUSCATE) > 0:
L2type = "MUSCATE"
derniereDate = d
So then I tried to find out which folders it is searching in the above code and which folders are available and I found it is searching for:
'../site_name/T33VWC/GIPP_001//S2?_OPER_SSC_L2VALD_T33VWC____20180601.DBL.DIR' (Natif)
and ../site_name/T33VWC/GIPP_001//SENTINEL2?_20180601-*_TT33VWC_C_V*(MUSCATE) while only SENTINEL2A_20180601-102024-463_L2A_T33VWC_C_V1-0 was in this folder (I don't get where the second T in TT33VWC instead of T33VWC is coming from.
What do you think is the best approaching to debugging this issue?
I know this is all very complicated and explained poorly, but I hope there is a simple solution for all this. If you need any more information on my setup please ask!
When I'm trying to install MAJA 3.3.2_TM on my openSUSE Leap 15.1 I received this error
Verifying archive integrity... 100% Error in MD5 checksums: d41d8cd98f00b204e9800998ecf8427e is different from 132ddb47e68a365ea7e12b8c837a71e9
I'm not sure, what cause this error, but a quick search suggests that the bash file is corrupted !
Do you have any suggestion, How to fix it ?
Hello,
We have noticed a file AUX_ECMWFT in GRANULE/L1C_T31TDM_A012947_20190829T105030/AUX_DATA within the zip file of a sentinel-2 sat image.
Is there a way to use this file's data as cam data?
Thanks and regards
After processing several tiles successfully I now get the following error. Any help would be much appreciated!:
2019-04-01T11:25:54.436522 D23515 maja-processing-3.2.2 3.2 [000000005439] [P] Starting GIP_L2DIRT reading ...
2019-04-01T11:25:54.453205 D23515 maja-processing-3.2.2 3.2 [000000005439] [P] Starting GIP_L2DIFT reading ...
2019-04-01T11:25:54.456985 D23515 maja-processing-3.2.2 3.2 [000000005439] [P] Starting GIP_L2ALBD reading ...
2019-04-01T11:25:54.473978 D23515 maja-processing-3.2.2 3.2 [000000005439] [P] Starting GIP_L2TOCR reading ...
2019-04-01T11:25:54.633096 D23515 maja-processing-3.2.2 3.2 [000000005439] [P] Starting L2Processor::PreProcessing() done.
2019-04-01T11:25:54.645610 D23515 maja-processing-3.2.2 3.2 [000000005439] [I] L2NominalProcessor::PreProcessing() done.
2019-04-01T11:25:54.662481 D23515 maja-processing-3.2.2 3.2 [000000005439] [I] Starting L2NominalProcessor::ScientificProcessing() ...
2019-04-01T11:25:54.680911 D23515 maja-processing-3.2.2 3.2 [000000005439] [I] The GIP_L2SITE file detected for the satellite 'SENTINEL-2_' is </media/au173212/bigdrive/JEM/Orkideer/temp_maja/Denmark/32VMJ/GIPP_MAJA_3.1.2_TM/in/S2__TEST_GIP_L2SITE_S_31TJF____10001_00000000_99999999.EEF>.
2019-04-01T11:25:54.695893 D23515 maja-processing-3.2.2 3.2 [000000005439] [I] Start Sentinel2 L1 ImageFileReader ...
2019-04-01T11:28:06.902702 D23515 maja-processing-3.2.2 3.2 [000000005439] [E] ****************************************************************************************************** [vnsMajaMainProcessor.cxx:main:130]
2019-04-01T11:28:06.903101 D23515 maja-processing-3.2.2 3.2 [000000005439] [E] ScientificProcessing: ITK Exception: itk::ERROR: MultiThreader(0x275e850): Exception occurred during SingleMethodExecute
/home/qt/setg_install/MACCS/06_Install/RH6/maja/core/3.2.2/plugin_tm_only/build/ITK/source/Modules/Core/Common/src/itkMultiThreaderPThreads.cxx:241:
itk::ERROR: MultiThreader(0x275e850): Unable to join thread. [itkMultiThreader.cxx:416] [vnsMajaMainProcessor.cxx:main:130]
2019-04-01T11:28:06.916093 D23515 maja-processing-3.2.2 3.2 [000000005439] [E] [vnsMajaMainProcessor.cxx:main:130]
2019-04-01T11:28:06.929948 D23515 maja-processing-3.2.2 3.2 [000000005439] [E] ****************************************************************************************************** [vnsMajaMainProcessor.cxx:main:130]
Hello,
I have a new issue on tile T29SNA concerning SWBD data.
According to logs of Maja, we need these tiles :
['w010n36', 'w010n37', 'w009n36', 'w009n37', 'w008n36', 'w008n37']
When I go to EarthExplorer and put the shapefile of T29SNA tile, the query returns only these SWBD tiles : 'w009n36', 'w009n37', 'w008n36', 'w008n37'
Furthermore, it seems that 'w010n36' 'w010n37' does not exist...
If you need more informations, don't hesitate.
Thanks a lot.
Nicolas
When I use download_CAMS.py or download_CAMS_daily.py, I get ERROR 6 (MARS_EXPECTED_FIELDS): Expected 155, got 0' when downloading CAMS data for March-June 2016. For 2017-2019 all the data has been downloading well.
Bug in the creation of a workplan with 12 products found but only 1 set for backward.
File "/softs/projets/maccs/MAJA_INSTALL/4.1.0/lib/python/orchestrator/processor/l2_backward_processor.py", line 93, in scientific_processing raise MajaDataException("L2BackwardProcessing need more than one input L1 image product.")
Workplan:
2020-03-02 17:30:58,858 [INFO ] 12 workplan(s) successfully created:
DATE | TILE | MODE | L1-PRODUCT | ADDITIONAL INFO
2018-04-03 10:40:19 | 31TFJ | BACKWARD | S2B_MSIL1C_20180403T104019_N0206_R008_T31TFJ_20180403T132754.SAFE | Backward of 1 products
2018-04-05 10:30:21 | 31TFJ | NOMINAL | S2A_MSIL1C_20180405T103021_N0206_R108_T31TFJ_20180405T141122.SAFE | L2 from previous
2018-04-08 10:40:21 | 31TFJ | NOMINAL | S2A_MSIL1C_20180408T104021_N0206_R008_T31TFJ_20180408T124948.SAFE | L2 from previous
2018-04-10 10:30:19 | 31TFJ | NOMINAL | S2B_MSIL1C_20180410T103019_N0206_R108_T31TFJ_20180410T131717.SAFE | L2 from previous
2018-04-13 10:40:19 | 31TFJ | NOMINAL | S2B_MSIL1C_20180413T104019_N0206_R008_T31TFJ_20180413T115514.SAFE | L2 from previous
2018-04-15 10:30:21 | 31TFJ | NOMINAL | S2A_MSIL1C_20180415T103021_N0206_R108_T31TFJ_20180415T142301.SAFE | L2 from previous
2018-04-18 10:40:21 | 31TFJ | NOMINAL | S2A_MSIL1C_20180418T104021_N0206_R008_T31TFJ_20180418T125356.SAFE | L2 from previous
2018-04-20 10:30:19 | 31TFJ | NOMINAL | S2B_MSIL1C_20180420T103019_N0206_R108_T31TFJ_20180420T114307.SAFE | L2 from previous
2018-04-23 10:40:19 | 31TFJ | NOMINAL | S2B_MSIL1C_20180423T104019_N0206_R008_T31TFJ_20180423T114954.SAFE | L2 from previous
2018-04-25 10:30:21 | 31TFJ | NOMINAL | S2A_MSIL1C_20180425T103021_N0206_R108_T31TFJ_20180425T174016.SAFE | L2 from previous
2018-04-28 10:40:21 | 31TFJ | NOMINAL | S2A_MSIL1C_20180428T104021_N0206_R008_T31TFJ_20180428T125712.SAFE | L2 from previous
2018-04-30 10:30:19 | 31TFJ | NOMINAL | S2B_MSIL1C_20180430T103019_N0206_R108_T31TFJ_20180430T113518.SAFE | L2 from previous
I have been working a bit with MAJA v3.3, and I have had a small problem with start_maja. When there is an image that is too cloudy the execution stops throwing the following error
Traceback (most recent call last):
File "./start_maja.py", line 655, in <module>
start_maja(folder_file, gipp, lut, site, tile, orbit, nb_backward, options, debug_mode)
File "./start_maja.py", line 555, in start_maja
valid = test_valid_L2A(nomL2init_MUSCATE[0])
File "./start_maja.py", line 226, in test_valid_L2A
os.rename(L2A_DIR, dir_name+"/L2NOTV_"+prod_name)
OSError: [Errno 13] Permission denied
I think this happens within the function test_valid_L2A because it tries to rename the parent directory while the metadata is still open by python, (JPIfile).
In my case I modified the function like this to avoid the error:
def test_valid_L2A(L2A_DIR):
# test validity of a Level2A product of MUSCATE type
JPIfile = glob.glob("%s/DATA/*_JPI_ALL.xml" % L2A_DIR.replace('[', '[[]'))[0]
valid = True
try:
with open(JPIfile) as f:
for ligne in f:
if ligne.find("<Value>L2NOTV</Value>") >= 0:
valid = False
if valid is False:
prod_name = os.path.basename(L2A_DIR)
dir_name = os.path.dirname(L2A_DIR)
if not(os.path.exists(dir_name+"/L2NOTV_"+prod_name)):
os.rename(L2A_DIR, dir_name+"/L2NOTV_"+prod_name)
else:
shutil.rmtree(dir_name+"/L2NOTV_"+prod_name)
os.rename(L2A_DIR, dir_name+"/L2NOTV_"+prod_name)
print("!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!")
print(
"L2A product %s is not valid (probably due to too many clouds or No_data values)" % dir_name)
print("!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!")
except IOError:
valid = False
print("!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!")
print("L2A product %s not found " % L2A_DIR)
print("!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!")
return(valid)
Hope this is useful,
Ignacio
DTM/S2__TEST_AUX_REFDE2_T31TFJ_0001 containing
S2__TEST_AUX_REFDE2_T31TFJ_0001.DBL.DIR
S2__TEST_AUX_REFDE2_T31TFJ_0001.HDR
These are the result of the 'Create DTM' step applied on L1C data
The L1C data was taken by performing
python ./peps_download.py -c S2ST -l --lon 4.8 --lat 43.97 -a peps.txt -d 2015-11-01 -f 2015-12-01 -w /var/local/maja/L1C_DATA/4-8_43-97
The folder /var/local/maja/L1C_DATA/4-8_43-97 contains
S2A_MSIL1C_20151117T103312_N0204_R108_T31TFJ_20151117T103313.SAFE
S2A_MSIL1C_20151117T103312_N0204_R108_T31TFJ_20151117T103313.zip
S2A_MSIL1C_20151120T104332_N0204_R008_T31TFJ_20151120T104750.SAFE
S2A_MSIL1C_20151120T104332_N0204_R008_T31TFJ_20151120T104750.zip
S2A_MSIL1C_20151127T103352_N0204_R108_T31TFJ_20151127T103440.SAFE
S2A_MSIL1C_20151127T103352_N0204_R108_T31TFJ_20151127T103440.zip
S2A_MSIL1C_20151130T104412_N0204_R008_T31TFJ_20151130T104646.SAFE
S2A_MSIL1C_20151130T104412_N0204_R008_T31TFJ_20151130T104646.zip
GIPP_S2AS2B/SENTINEL2_TM/ containing files S2B_TEST_GIP_CKEXTL_S_ALLSITES_00001_20190626_21000101.EEF
S2A_TEST_GIP_CKEXTL_S_ALLSITES_00001_20190626_21000101.EEF S2B_TEST_GIP_CKEXTL_S_ALLSITES_10001_20190626_21000101.EEF
S2A_TEST_GIP_CKEXTL_S_ALLSITES_10001_20190626_21000101.EEF S2B_TEST_GIP_CKQLTL_S_ALLSITES_00001_20190626_21000101.EEF
S2A_TEST_GIP_CKQLTL_S_ALLSITES_00001_20190626_21000101.EEF S2B_TEST_GIP_CKQLTL_S_ALLSITES_10001_20190626_21000101.EEF
S2A_TEST_GIP_CKQLTL_S_ALLSITES_10001_20190626_21000101.EEF S2B_TEST_GIP_L2COMM_L_ALLSITES_00001_20190626_21000101.EEF
S2A_TEST_GIP_L2COMM_L_ALLSITES_00001_20190626_21000101.EEF S2B_TEST_GIP_L2COMM_L_ALLSITES_10001_20190626_21000101.EEF
S2A_TEST_GIP_L2COMM_L_ALLSITES_10001_20190626_21000101.EEF S2B_TEST_GIP_L2SMAC_L_ALLSITES_00001_20190626_21000101.EEF
S2A_TEST_GIP_L2SMAC_L_ALLSITES_00001_20190626_21000101.EEF S2__TEST_GIP_L2SITE_S_ALLSITES_00001_20190626_21000101.EEF
LUT_SENT2_NOCAMS/LUTs containing (https://zenodo.org/record/3368623)
S2A_TEST_GIP_L2ALBD_L_CONTINEN_00001_20190626_21000101.DBL.DIR S2B_TEST_GIP_L2ALBD_L_CONTINEN_00001_20190626_21000101.DBL.DIR
S2A_TEST_GIP_L2ALBD_L_CONTINEN_00001_20190626_21000101.HDR S2B_TEST_GIP_L2ALBD_L_CONTINEN_00001_20190626_21000101.HDR
S2A_TEST_GIP_L2DIFT_L_CONTINEN_00001_20190626_21000101.DBL.DIR S2B_TEST_GIP_L2DIFT_L_CONTINEN_00001_20190626_21000101.DBL.DIR
S2A_TEST_GIP_L2DIFT_L_CONTINEN_00001_20190626_21000101.HDR S2B_TEST_GIP_L2DIFT_L_CONTINEN_00001_20190626_21000101.HDR
S2A_TEST_GIP_L2DIRT_L_CONTINEN_00001_20190626_21000101.DBL.DIR S2B_TEST_GIP_L2DIRT_L_CONTINEN_00001_20190626_21000101.DBL.DIR
S2A_TEST_GIP_L2DIRT_L_CONTINEN_00001_20190626_21000101.HDR S2B_TEST_GIP_L2DIRT_L_CONTINEN_00001_20190626_21000101.HDR
S2A_TEST_GIP_L2TOCR_L_CONTINEN_00001_20190626_21000101.DBL.DIR S2B_TEST_GIP_L2TOCR_L_CONTINEN_00001_20190626_21000101.DBL.DIR
S2A_TEST_GIP_L2TOCR_L_CONTINEN_00001_20190626_21000101.HDR S2B_TEST_GIP_L2TOCR_L_CONTINEN_00001_20190626_21000101.HDR
S2A_TEST_GIP_L2WATV_L_CONTINEN_00001_20190626_21000101.DBL.DIR S2B_TEST_GIP_L2WATV_L_CONTINEN_00001_20190626_21000101.DBL.DIR
S2A_TEST_GIP_L2WATV_L_CONTINEN_00001_20190626_21000101.HDR S2B_TEST_GIP_L2WATV_L_CONTINEN_00001_20190626_21000101.HDR
TMP_FILES/
This is the folder.txt file:
repCode=/var/local/maja/Start-MAJA
repWork=/var/local/maja/Start-MAJA/TMP_FILES
repL1 =/var/local/maja/L1C_DATA
repL2 =/var/local/maja/L2A_DATA
repMaja=/var/local/maja/bin/maja
repCAMS=/var/local/maja/Start-MAJA/CAMS
I encounter an error when performing start_maja.py
python ./start_maja.py -f folders.txt -g GIPP_S2AS2B/SENTINEL2_TM -l LUT_SENT2_NOCAMS/LUTs -t 31TFJ -s 4-8_43-97 -d 20151101 -e 20151201
2020-04-02 15:59:24,574 - Start-Maja - INFO - /var/local/maja/L2A_DATA/4-8_43-97/31TFJ/GIPP_S2AS2B/SENTINEL2_TM//S2?_OPER_SSC_L2VALD_31TFJ____T20151117.DBL.DIR not found
2020-04-02 15:59:24,574 - Start-Maja - INFO - /var/local/maja/L2A_DATA/4-8_43-97/31TFJ/GIPP_S2AS2B/SENTINEL2_TM//SENTINEL2?_20151117-_T31TFJ_C_V not found
2020-04-02 15:59:24,575 - Start-Maja - INFO - /var/local/maja/L2A_DATA/4-8_43-97/31TFJ/GIPP_S2AS2B/SENTINEL2_TM//S2?_OPER_SSC_L2VALD_31TFJ____T20151120.DBL.DIR not found
2020-04-02 15:59:24,575 - Start-Maja - INFO - /var/local/maja/L2A_DATA/4-8_43-97/31TFJ/GIPP_S2AS2B/SENTINEL2_TM//SENTINEL2?_20151120-_T31TFJ_C_V not found
2020-04-02 15:59:24,576 - Start-Maja - INFO - /var/local/maja/L2A_DATA/4-8_43-97/31TFJ/GIPP_S2AS2B/SENTINEL2_TM//S2?_OPER_SSC_L2VALD_31TFJ____T20151127.DBL.DIR not found
2020-04-02 15:59:24,576 - Start-Maja - INFO - /var/local/maja/L2A_DATA/4-8_43-97/31TFJ/GIPP_S2AS2B/SENTINEL2_TM//SENTINEL2?_20151127-_T31TFJ_C_V not found
2020-04-02 15:59:24,577 - Start-Maja - INFO - /var/local/maja/L2A_DATA/4-8_43-97/31TFJ/GIPP_S2AS2B/SENTINEL2_TM//S2?_OPER_SSC_L2VALD_31TFJ____T20151130.DBL.DIR not found
2020-04-02 15:59:24,577 - Start-Maja - INFO - /var/local/maja/L2A_DATA/4-8_43-97/31TFJ/GIPP_S2AS2B/SENTINEL2_TM//SENTINEL2?_20151130-_T31TFJ_C_V not found
2020-04-02 15:59:24,577 - Start-Maja - INFO - No existing L2 product, we start with backward mode
2020-04-02 15:59:24,577 - Start-Maja - INFO - => processing date 20151117
2020-04-02 15:59:24,577 - Start-Maja - INFO - dates to process in backward mode :
2020-04-02 15:59:24,577 - Start-Maja - INFO - -- 20151117 : /var/local/maja/L1C_DATA/4-8_43-97/S2A_MSIL1C_20151117T103312_N0204_R108_T31TFJ_20151117T103313.SAFE
2020-04-02 15:59:24,577 - Start-Maja - INFO - -- 20151120 : /var/local/maja/L1C_DATA/4-8_43-97/S2A_MSIL1C_20151120T104332_N0204_R008_T31TFJ_20151120T104750.SAFE
2020-04-02 15:59:24,577 - Start-Maja - INFO - -- 20151127 : /var/local/maja/L1C_DATA/4-8_43-97/S2A_MSIL1C_20151127T103352_N0204_R108_T31TFJ_20151127T103440.SAFE
2020-04-02 15:59:24,577 - Start-Maja - INFO - -- 20151130 : /var/local/maja/L1C_DATA/4-8_43-97/S2A_MSIL1C_20151130T104412_N0204_R008_T31TFJ_20151130T104646.SAFE
2020-04-02 15:59:24,581 - Start-Maja - INFO - #################################
2020-04-02 15:59:24,581 - Start-Maja - INFO - #################################
2020-04-02 15:59:24,581 - Start-Maja - INFO - processing /var/local/maja/L1C_DATA/4-8_43-97/S2A_MSIL1C_20151117T103312_N0204_R108_T31TFJ_20151117T103313.SAFE in backward mode
2020-04-02 15:59:24,581 - Start-Maja - INFO - Initialisation mode with backward is longer
2020-04-02 15:59:24,581 - Start-Maja - INFO - MAJA logfile: /var/local/maja/L2A_DATA/4-8_43-97/31TFJ/GIPP_S2AS2B/SENTINEL2_TM//S2A_MSIL1C_20151117T103312_N0204_R108_T31TFJ_20151117T103313.SAFE.log
2020-04-02 15:59:24,582 - Start-Maja - INFO - #################################
2020-04-02 15:59:24,586 - Start-Maja - INFO - => processing date 20151120
2020-04-02 15:59:24,587 - Start-Maja - INFO - MAJA command failed : /var/local/maja/bin/maja -i /var/local/maja/Start-MAJA/TMP_FILES/4-8_43-97/31TFJ/GIPP_S2AS2B/SENTINEL2_TM//in -o /var/local/maja/L2A_DATA/4-8_43-97/31TFJ/GIPP_S2AS2B/SENTINEL2_TM/ -m L2BACKWARD -ucs /var/local/maja/Start-MAJA/TMP_FILES/4-8_43-97/31TFJ/GIPP_S2AS2B/SENTINEL2_TM//userconf --TileId 31TFJ >/var/local/maja/L2A_DATA/4-8_43-97/31TFJ/GIPP_S2AS2B/SENTINEL2_TM//S2A_MSIL1C_20151117T103312_N0204_R108_T31TFJ_20151117T103313.SAFE.log 2>&1
2020-04-02 15:59:24,587 - Start-Maja - ERROR - First backward processing was unsuccessful, check MAJA installation
I installed MAJA on a virtualBox Xubuntu with shared windows folders and I had an error when creating the symbolic link:
Traceback (most recent call last):
File "start_maja.py", line 662, in
start_maja(folder_file, gipp, lut, site, tile, orbit, nb_backward, options, debug_mode)
File "start_maja.py", line 431, in start_maja
add_config_files(repConf, repWork + "userconf")
File "start_maja.py", line 186, in add_config_files
os.symlink(repConf, repWorkConf)
OSError: [Errno 1] Operation not permitted
I solved it with a command on the windows host:
C:\Program Files\Oracle\VirtualBox>VBoxManage setextradata oo_unix VBoxInternal2/SharedFoldersEnableSymlinksCreate/D_DRIVE 1
and restarted VirtualBox.
I installed MAJA on a virtualBox Xubuntu with shared windows folders and I had an error when creating the symbolic link:
Traceback (most recent call last):
File "start_maja.py", line 662, in
start_maja(folder_file, gipp, lut, site, tile, orbit, nb_backward, options, debug_mode)
File "start_maja.py", line 431, in start_maja
add_config_files(repConf, repWork + "userconf")
File "start_maja.py", line 186, in add_config_files
os.symlink(repConf, repWorkConf)
OSError: [Errno 71] Protocol error
I solved it ... running virtualBox as an administrator
Hi there,
I am trying to run Maja on an area where no SRTM or SBWD data seems to be available (in Finland). Is it still possible to use the tool?
I tried running the program and I get the following error:
2019-07-25T10:56:26.906582 ip-172-31-46-119 maja-processing-3.3.0 3.3 [000000004774] [P] -----------------------------------------------------------------------------------------------
2019-07-25T10:56:26.906725 ip-172-31-46-119 maja-processing-3.3.0 3.3 [000000004774] [P] Starting L2Processor::PreProcessing() ...
2019-07-25T10:56:26.907257 ip-172-31-46-119 maja-processing-3.3.0 3.3 [000000004774] [I] List of the 2 satellite(s) activated :
2019-07-25T10:56:26.907774 ip-172-31-46-119 maja-processing-3.3.0 3.3 [000000004774] [I] - SENTINEL-2A ('common' satellite SENTINEL-2_ and plugin SENTINEL2_TM)
2019-07-25T10:56:26.908303 ip-172-31-46-119 maja-processing-3.3.0 3.3 [000000004774] [I] - SENTINEL-2B ('common' satellite SENTINEL-2_ and plugin SENTINEL2_TM)
2019-07-25T10:56:26.908801 ip-172-31-46-119 maja-processing-3.3.0 3.3 [000000004774] [I] Starting reading and checking input data common for each plugin activated...
2019-07-25T10:56:26.909387 ip-172-31-46-119 maja-processing-3.3.0 3.3 [000000004774] [I] According to the input data, there is(are) 1 plugin(s) activated:
2019-07-25T10:56:26.909980 ip-172-31-46-119 maja-processing-3.3.0 3.3 [000000004774] [I] - SENTINEL2_TM
2019-07-25T10:56:26.910640 ip-172-31-46-119 maja-processing-3.3.0 3.3 [000000004774] [I] - SENTINEL2_TM
2019-07-25T10:56:27.178671 ip-172-31-46-119 maja-processing-3.3.0 3.3 [000000004774] [E] vns::Data::ERROR: DataApplicationHandler(0xc667f0): In the input directory, there is no GIPP input data matching with the 'AUX_REFDE2' File_Type pattern! [vnsDataApplicationHandler.cxx:GetListOfGippFilenames:1699]
2019-07-25T10:56:27.183059 ip-172-31-46-119 maja-processing-3.3.0 3.3 [000000004774] [E] ****************************************************************************************************** [vnsMajaMainProcessor.cxx:main:129]
2019-07-25T10:56:27.183178 ip-172-31-46-119 maja-processing-3.3.0 3.3 [000000004774] [E] PreProcessing: vns::Data::ERROR: DataApplicationHandler(0xc667f0): In the input directory, there is no GIPP input data matching with the 'AUX_REFDE2' File_Type pattern! [vnsDataApplicationHandler.cxx:1699] [MAJA Data Exception] [vnsMajaMainProcessor.cxx:main:129]
2019-07-25T10:56:27.183780 ip-172-31-46-119 maja-processing-3.3.0 3.3 [000000004774] [E] [vnsMajaMainProcessor.cxx:main:129]
2019-07-25T10:56:27.184357 ip-172-31-46-119 maja-processing-3.3.0 3.3 [000000004774] [E] ****************************************************************************************************** [vnsMajaMainProcessor.cxx:main:129]
For two tiles (33UYQ, 32TQT), which were correctly processed before, MAJA fails now. It does not proceed, it loops and the size of PMC_LxREPT.EEF continuously increases. Other tiles are still processed correctly.
MAJA Version 3.3.0
Error message: vns::Business::ERROR: ComputeScatteringCorrectionImageFilter(0x20cf500): For band id '3' (nb channel='4') ' no miniLUT has been generated for this angle zone (detector) : '4'. Angles zones are [0,5,6,7,8,9,10,] !!!,Check coherency between metadata and input zone mask source. [vnsComputeScatteringCorrectionImageFilter.txx:ThreadedGenerateData:238]
Last correctly processed product: S2A_MSIL1C_20200310T094031_N0209_R036_T33UYQ_20200310T101526.SAFE
Product where MAJA fails: S2A_MSIL1C_20200313T095031_N0209_R079_T33UYQ_20200313T102505.SAFE
and does not further continue.
Log-file for the product where MAJA fails:
S2A_MSIL1C_20200313T095031_N0209_R079_T33UYQ_20200313T102505.SAFE.log
I am having trouble running Start-MAJA on some data. I have tried to put the directories together according to the instructions, but I get the following error message when running the Python script:
$ python /mnt/passesData/RunMAJA/start_maja.py -v -f /mnt/passesData/RunMAJA/folders.txt -c GIPP_MAJA_3.1.2_MUSCATE -t 29TPH -s Spain -d 20180823 -e 20180902
2019-03-11 11:53:43,327 - Start-Maja - INFO - No existing L2 product, we start with backward mode
2019-03-11 11:53:43,327 - Start-Maja - INFO - => processing date 20180823
2019-03-11 11:53:43,327 - Start-Maja - INFO - dates to process in backward mode :
2019-03-11 11:53:43,327 - Start-Maja - INFO - -- 20180823 : /mnt/passesData/MAJAConfFold/L1C_PDGS/Spain/S2B_MSIL1C_20180823T112109_N0206_R037_T29TPH_20180823T152254.SAFE
2019-03-11 11:53:43,327 - Start-Maja - INFO - -- 20180828 : /mnt/passesData/MAJAConfFold/L1C_PDGS/Spain/S2A_MSIL1C_20180828T112111_N0206_R037_T29TPH_20180828T135154.SAFE
2019-03-11 11:53:43,327 - Start-Maja - INFO - -- 20180902 : /mnt/passesData/MAJAConfFold/L1C_PDGS/Spain/S2B_MSIL1C_20180902T112109_N0206_R037_T29TPH_20180902T140408.SAFE
Traceback (most recent call last):
File "/mnt/passesData/RunMAJA/start_maja.py", line 613, in <module>
start_maja(folder_file, context, site, tile, orbit, nb_backward, options, debug_mode)
File "/mnt/passesData/RunMAJA/start_maja.py", line 449, in start_maja
add_parameter_files(repGipp, repWork + "/in/", tile, repCams)
File "/mnt/passesData/RunMAJA/start_maja.py", line 158, in add_parameter_files
os.symlink(fic, repWorkIn + '/' + base)
OSError: [Errno 17] File exists
The folders.txt file contains:
repCode=/mnt/passesData/RunMAJA
repWork=/mnt/passesData/MAJAConfFold/MAJAtemp
repL1 =/mnt/passesData/MAJAConfFold/L1C_PDGS
repL2 =/mnt/passesData/MAJAConfFold/L2A_MAJA
repMaja=/opt/maja/3.2.2/bin/maja
repCAMS=/mnt/passesData/MAJAConfFold/CAMS
It looks like it is trying to create the links in the repWorkIn directory twice. If I look in that directory after it has failed, it contains links to the three Sentinel-2 SAFE folders I want to process. If I delete those links myself and run it again, it gives the same error.
I can provide any other information you may need. Thank you very much.
I am trying to run MAJA 3.3.0 without CAMS data for the test region around RIO, but am blocked by the following :
S2A_MSIL1C_20160802T130252_N0204_R095_T23KPQ_20160802T130656.SAFE.log
The GIPP directory used is GIPP_S2_MAJA_3.3_TM,downloaded from http://tully.ups-tlse.fr/olivier/gipp_maja/tree/master/GIPP_S2_MAJA_3.3_TM.The GIPP file was in the directory of start-maja.py .Here is the folder.txt that I configure:
folders.txt
Can you help me? I really need your help. Thank you!
I am trying to run MAJA 3.3.0 without CAMs data for the test region around Avignon, but am blocked by the following error:
vns::Data::ERROR: DataApplicationHandler(0xa235d0): No GIPP of type GIP_CKEXTL has been detected for the Mission in the input directory! [vnsDataApplicationHandler.cxx:1501] [MAJA Data Exception] [vnsMajaMainProcessor.cxx:main:131]
I downloaded the additional look-up tables from https://zenodo.org/record/2636694 though CAMs should be deactivated, but either way I get blocked by the error.
Any help on how to proceed would be great, thanks!
Hello Olivier,
I hope you are enjoying your vacation. Gonzalo and I have had a chat to update me what happened during my absence. MAJA is still causing several issues for us.
One of them is the processing time and the space required. I tried to run it on one month's worth of data (9 images) and we provided around 30GB of disc space, which wasn't sufficient. It produced an error after over 1 hour running time due to lack of space. Could you give us an estimation how much disk space and time is needed for processing one tile? Alternatively, is there a way to reduce the disk space needed for the processing?
Secondly, since we want to investigate a larger region, we aim at reusing the flat surface reflectances so we don't have to go through the entire multi-temporal processing chain every time a new image for a tile comes in. Could you please give us an idea how we can reuse it and how it will affect the processing parameters (disk space/ time)?
Thanks for your help!
Daria
Originally posted by @DariaLudtke in #24 (comment)
Hi Oliver, thanks for looking at the SWBD issue.
The initial fix to handle SWBD files that are out of bounds (don't exits) worked to resolve the first issue extracting the SHP files but the prepare_dtm.py routine fails during gdal_rasterize. It looks like the unavailable SWBD files are added to the list populated in 170-188 of tuilage_mnt_eau_S2.py includes the unavailable files. Enumerating the list on 506 of lib_mnt.py triggers the error print('missing SWBD watr file : ', [racine_nom_eau).] during the gdal_rasterize phase.
The script I execute is on tile 60HXD:
python Start-MAJA/prepare_dtm/DTMCreation.py -k Start-MAJA/prepare_dtm/S2Tiling.kml -g 60HXD -w Start-MAJA/prepare_dtm/Water/ -s Start-MAJA/prepare_dtm/SRTM/ -o Start-MAJA/prepare_dtm/DTM/60HXD
The results is:
> Found SRTM zip-archives...
> Found Water zip-archives...
> Processing with GranuleID
> 60HXD
> UTM60S
> 32760
> EPSG:32760
> 0
> 0
> 0
> 0
> 109800
> 109800
> 0
> 600000
> 5900020
> Working directory: /tmp/60HXDwj66g3zk
> Start-MAJA/prepare_dtm/SRTM/ /tmp/60HXDwj66g3zk Start-MAJA/prepare_dtm/Water/ /tmp/60HXDwj66g3zk
> [72, 20]
> [72, 20]
> (178.1244775712774, -37.040713289972636, 0.0) (179.38979675735044, -38.01144337672406, 0.0)
> [72, 20] [72, 20]
> ['srtm_72_20.tif']
> (178.1244775712774, -37.040713289972636, 0.0) (179.38979675735044, -38.01144337672406, 0.0)
> [178, -38] [179, -39]
> longitudes 178 179
> latitudes -39 -38
> center coordinates [[178.5, -38.5], [178.5, -37.5], [179.5, -38.5], [179.5, -37.5]]
> ['e178s39', 'e178s38', 'e179s39', 'e179s38']
> /tmp/60HXDwj66g3zk
> liste_fic_mnt ['srtm_72_20.tif']
> Archive: Start-MAJA/prepare_dtm/Water/e178s39i.zip
> inflating: /tmp/60HXDwj66g3zk/tmpbsclecsi/e178s39i.dbf
> inflating: /tmp/60HXDwj66g3zk/tmpbsclecsi/e178s39i.shp
> inflating: /tmp/60HXDwj66g3zk/tmpbsclecsi/e178s39i.shx
> creating: /tmp/60HXDwj66g3zk/tmpbsclecsi/Documents/
> inflating: /tmp/60HXDwj66g3zk/tmpbsclecsi/Documents/Readme File SRTM Water Body Data.doc
> inflating: /tmp/60HXDwj66g3zk/tmpbsclecsi/Documents/SRTM Edit Rules v2.0 12 Mar 03.doc
> inflating: /tmp/60HXDwj66g3zk/tmpbsclecsi/Documents/SWBD Product Specific Guidance v2.0 12 Mar 03jas.doc
> Archive: Start-MAJA/prepare_dtm/Water/e178s38i.zip
> inflating: /tmp/60HXDwj66g3zk/tmpbsclecsi/e178s38i.dbf
> inflating: /tmp/60HXDwj66g3zk/tmpbsclecsi/e178s38i.shp
> inflating: /tmp/60HXDwj66g3zk/tmpbsclecsi/e178s38i.shx
> inflating: /tmp/60HXDwj66g3zk/tmpbsclecsi/Documents/Readme File SRTM Water Body Data.doc
> inflating: /tmp/60HXDwj66g3zk/tmpbsclecsi/Documents/SRTM Edit Rules v2.0 12 Mar 03.doc
> inflating: /tmp/60HXDwj66g3zk/tmpbsclecsi/Documents/SWBD Product Specific Guidance v2.0 12 Mar 03jas.doc
> FIC: srtm_72_20.tif
> <class 'str'> <class 'str'>
> Start-MAJA/prepare_dtm/SRTM//srtm_72_20.tif
> Archive: Start-MAJA/prepare_dtm/SRTM//srtm_72_20.zip
> inflating: /tmp/60HXDwj66g3zk/tmpbsclecsi/readme.txt
> inflating: /tmp/60HXDwj66g3zk/tmpbsclecsi/srtm_72_20.hdr
> inflating: /tmp/60HXDwj66g3zk/tmpbsclecsi/srtm_72_20.tfw
> inflating: /tmp/60HXDwj66g3zk/tmpbsclecsi/srtm_72_20.tif
> gdalwarp -r cubic -srcnodata -32767 -dstnodata 0 /tmp/60HXDwj66g3zk/tmpbsclecsi/srtm_72_20.tif /tmp/60HXDwj66g3zk/tmpbsclecsi/srtm_72_20nodata0.tif
>
> Creating output file that is 6001P x 6001L.
> Processing input file /tmp/60HXDwj66g3zk/tmpbsclecsi/srtm_72_20.tif.
> 0...10...20...30...40...50...60...70...80...90...100 - done.
> e178s39
> #############Fichier eau : /tmp/60HXDwj66g3zk/tmpbsclecsi/e178s39i.shp
> gdal_rasterize -burn 1 -l e178s39i /tmp/60HXDwj66g3zk/tmpbsclecsi/e178s39i.shp /tmp/60HXDwj66g3zk/tmpbsclecsi/srtm_72_20_tmp.tif
> Warning 1: The output raster dataset has a SRS, but the input vector layer SRS is unknown.
> Ensure input vector has the same SRS, otherwise results might be incorrect.
> 0...10...20...30...40...50...60...70...80...90...100 - done.
> e178s38
> #############Fichier eau : /tmp/60HXDwj66g3zk/tmpbsclecsi/e178s38i.shp
> gdal_rasterize -burn 1 -l e178s38i /tmp/60HXDwj66g3zk/tmpbsclecsi/e178s38i.shp /tmp/60HXDwj66g3zk/tmpbsclecsi/srtm_72_20_tmp.tif
> Warning 1: The output raster dataset has a SRS, but the input vector layer SRS is unknown.
> Ensure input vector has the same SRS, otherwise results might be incorrect.
> 0...10...20...30...40...50...60...70...80...90...100 - done.
> e179s39
> missing SWBD watr file : e179s39
> Traceback (most recent call last):
> File "Start-MAJA/prepare_dtm/DTMCreation.py", line 326, in <module>
> creator.run(args.out, args.tempout)
> File "Start-MAJA/prepare_dtm/DTMCreation.py", line 286, in run
> water_zipped = self.WaterZipped)
> File "Start-MAJA/prepare_dtm/../prepare_dtm/tuilage_mnt_eau_S2.py", line 203, in run
> calcul_masque_eau_mnt, working_dir=working_dir)
> File "Start-MAJA/prepare_dtm/../prepare_dtm/lib_mnt.py", line 514, in fusion_mnt
> land = TestLand(liste_centre_eau[i][0], liste_centre_eau[i][1])
> File "Start-MAJA/prepare_dtm/../prepare_dtm/lib_mnt.py", line 57, in TestLand
> layer = dataSource.GetLayer()
> AttributeError: 'NoneType' object has no attribute 'GetLayer'
Thanks
Hello,
I am currently running MAJA with CAMS on a cloudy dataset (one month is specifically cloudy).
I am having the following log in the console:
2019-11-25 13:22:55,116 - Start-Maja - INFO - => processing date 20190712 2019-11-25 13:22:55,213 - Start-Maja - INFO - Using MUSCATE L2 type 2019-11-25 13:22:55,215 - Start-Maja - INFO - previous L2 : /workspace/workdir/L2A//36UVB/GIPP_S2AS2B_xxx/SENTINEL2_TM_CAMS/SENTINEL2A_20190702-090611-775_L2A_T36UVB_C_V1-0 2019-11-25 13:22:57,044 - Start-Maja - INFO - ################################# 2019-11-25 13:22:57,044 - Start-Maja - INFO - ################################# 2019-11-25 13:22:57,044 - Start-Maja - INFO - processing /media/scratch/MAJA/L1C/36UVB/S2A_MSIL1C_20190712T085601_N0208_R007_T36UVB_20190712T110428.zip in nominal mode 2019-11-25 13:22:57,044 - Start-Maja - INFO - MAJA logfile: /workspace/workdir/L2A//36UVB/GIPP_S2AS2B_xxx/SENTINEL2_TM_CAMS//S2A_MSIL1C_20190712T085601_N0208_R007_T36UVB_20190712T110428.zip.log 2019-11-25 13:22:57,045 - Start-Maja - INFO - ################################# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! L2A product /workspace/workdir/L2A//36UVB/GIPP_S2AS2B_xxx/SENTINEL2_TM_CAMS is not valid (probably due to too many clouds or No_data values) !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
My questions are the following:
There is an error on the building of path during extraction of .zip file.
Actually, the extraction create a folder with the same name of the .zip, and when you make a merge (line 486 for example), this folder is forgotten. Thus, the script does not find the right file.
You can see below the console return :
ERROR :
/mnt/data_netapp/raster/monde/Perou/SRTM/srtm_21_14.tif
Archive: /mnt/data_netapp/raster/monde/Perou/SRTM/srtm_21_14.zip
creating: /tmp/18MUSp0z_hpbk/tmp063det8b/srtm_21_14/
inflating: /tmp/18MUSp0z_hpbk/tmp063det8b/srtm_21_14/srtm_21_14.tif.aux.xml
inflating: /tmp/18MUSp0z_hpbk/tmp063det8b/srtm_21_14/readme.txt
inflating: /tmp/18MUSp0z_hpbk/tmp063det8b/srtm_21_14/srtm_21_14.hdr
inflating: /tmp/18MUSp0z_hpbk/tmp063det8b/srtm_21_14/srtm_21_14.tfw
inflating: /tmp/18MUSp0z_hpbk/tmp063det8b/srtm_21_14/srtm_21_14.tif
gdalwarp -r cubic -srcnodata -32767 -dstnodata 0 /tmp/18MUSp0z_hpbk/tmp063det8b/srtm_21_14.tif /tmp/18MUSp0z_hpbk/tmp063det8b/srtm_21_14nodata0.tif
ERROR 4: /tmp/18MUSp0z_hpbk/tmp063det8b/srtm_21_14.tif: No such file or directory
ERROR 4: /tmp/18MUSp0z_hpbk/tmp063det8b/srtm_21_14.tif: No such file or directory
Hello,
I have a problem when I run the DTMcreation.py script. The error messages reads: Cannot find SWBD files!
I believe I added correctly the address of the directory containing SRTM water body data, yet this error persist!
I also tried unpacking the data into the directory and also just leaving the zip filed, but in both cases I got the same error.
Attached is the full error message
Fails to get DTM for tile 32TPR: error on jp2 file format raises an error in Common/ImageIO.py
2020-03-11 16:42:48,842 [DEBUG] Searching for DTM
2020-03-11 16:42:48,847 [DEBUG] Cannot find DTM. Will attempt to download it...
2020-03-11 16:42:48,847 [INFO ] Attempting to download DTM...
ERROR 4: /work/datalake/S2-L1C/32TPR/2018/05/02/S2A_MSIL1C_20180502T102031_N0206_R065_T32TPR_20180502T154426.SAFE/GRANULE/L1C_T32TPR_A014934_20180502T102434/IMG_DATA/T32TPR_20180502T102031_B02.jp2 not recognized as a supported file format.
Traceback (most recent call last):
File "Start_maja.py", line 567, in <module>
s.run()
File "Start_maja.py", line 497, in run
self.avail_input_l1[0].get_mnt(dem_dir=self.rep_mnt)
File "/work/scratch/colinj/sandbox/Start-MAJA/Chain/Product.py", line 187, in get_mnt
return MNTFactory(site=self.mnt_site, platform_id=self.platform_str,
File "/work/scratch/colinj/sandbox/Start-MAJA/Chain/S2Product.py", line 71, in mnt_site
return Site.from_raster(self.tile, band_b2)
File "/work/scratch/colinj/sandbox/Start-MAJA/prepare_mnt/mnt/SiteInfo.py", line 83, in from_raster
raster, driver = ImageIO.tiff_to_array(raster, array_only=False)
File "/work/scratch/colinj/sandbox/Start-MAJA/Common/ImageIO.py", line 41, in tiff_to_array
tiff_array = np.array(gdo.ReadAsArray(lon_offset_px, lat_offset_px))
AttributeError: 'NoneType' object has no attribute 'ReadAsArray'
While trying to download the monthly CAMS data with the following command and my unprivileged account, I got an error (from ECMWF library). Does that mean I need a privileged account?
$ ./download_CAMS.py -d 20150601 -f 20190430 -k -w /data/PBA_gluster/projects/MAJA/CAMS/write -s ncks -k -a /data/PBA_gluster/projects/MAJA/CAMS/DBL
mars - INFO - 20190507.153712 - Maximum retrieval size is 30.00 G
retrieve,stream=oper,levelist=1/2/3/5/7/10/20/30/50/70/100/150/200/250/300/400/500/600/700/850/925/1000,area=g,levtype=pl,expver=0001,padding=0,step=0,grid=1.25/1.25,param=157.128,time=00,date=20150601/to/201506
30,type=fc,class=mcmars - INFO - 20190507.153712 - Automatic split by date is on
mars - INFO - 20190507.153712 - Processing request 1
RETRIEVE,
CLASS = MC,
TYPE = FC,
STREAM = OPER,
EXPVER = 0001,
REPRES = SH,
LEVTYPE = PL,
LEVELIST = 1/2/3/5/7/10/20/30/50/70/100/150/200/250/300/400/500/600/700/850/925/1000,
PARAM = 157.128,
TIME = 0000,
STEP = 0,
DOMAIN = G,
RESOL = AUTO,
GRID = 1.25/1.25,
PADDING = 0,
DATE = 20150601/20150602/20150603/20150604/20150605/20150606/20150607/20150608/20150609/20150610/20150611/20150612/20150613/20150614/20150615/20150616/20150617/20150618/20150619/20150620/20150621/20150
622/20150623/20150624/20150625/20150626/20150627/20150628/20150629/20150630
mars - INFO - 20190507.153712 - Web API request id: 5cd1a5a2a2de1951c1982e58
mars - INFO - 20190507.153712 - Requesting 660 fields
mars - INFO - 20190507.153712 - Calling mars on 'marser', callback on 55669
2019-05-07 17:37:23 Request is active
mars - INFO - 20190507.171224 - Server task is 65 [marser]
mars - INFO - 20190507.171225 - Request cost: 630 fields, 79.6813 Mbytes on 2 tapes, nodes: hpss [marser]
mars - INFO - 20190507.173430 - Transfering 83551860 bytes
mars - WARN - 20190507.173446 - Visiting database marser : expected 660, got 630
mars - INFO - 20190507.173446 - 630 fields have been interpolated
mars - ERROR - 20190507.173446 - Expected 660, got 630.
mars - ERROR - 20190507.173446 - Request failed
...
I run MAJA 3.2.2 on Ubuntu 16.04. I get this error in the beginning. However, after this error it seems to continue, but then gives another error (that may be linked to this one)
First error:
2019-03-15 14:33:10,116 - Start-Maja - INFO - => processing date 20160607
Traceback (most recent call last):
File "./start_maja.py", line 613, in <module>
start_maja(folder_file, context, site, tile, orbit, nb_backward, options, debug_mode)
File "./start_maja.py", line 468, in start_maja
logger.info("Using %s L2 type" % L2type)
UnboundLocalError: local variable 'L2type' referenced before assignment
Second error:
2019-03-15T14:33:10.804557 D23515 maja-processing-3.2.2 3.2 [000000028370] [I] The process uses 8 thread(s). Note: possible to use of maximum 128 threads.
2019-03-15T14:33:10.821243 D23515 maja-processing-3.2.2 3.2 [000000028370] [I] Starting L2BackwardProcessor::PreProcessing() ...
2019-03-15T14:33:10.830535 D23515 maja-processing-3.2.2 3.2 [000000028370] [E] vns::Data::ERROR: : Impossible to detect a L1 product in the directory </media/au173212/bigdrive/JEM/Orkideer/temp_maja/Denmark/32UMF/GIPP_MAJA_3.1.2_TM//in>. [vnsL1ImageInformationsProviderFactory.cxx:GetListOfL1Products:150]
2019-03-15T14:33:10.845801 D23515 maja-processing-3.2.2 3.2 [000000028370] [E] ****************************************************************************************************** [vnsMajaMainProcessor.cxx:main:129]
2019-03-15T14:33:10.854450 D23515 maja-processing-3.2.2 3.2 [000000028370] [E] PreProcessing: vns::Data::ERROR: : Impossible to detect a L1 product in the directory </media/au173212/bigdrive/JEM/Orkideer/temp_maja/Denmark/32UMF/GIPP_MAJA_3.1.2_TM//in>. [vnsL1ImageInformationsProviderFactory.cxx:150] [MAJA Data Exception] [vnsMajaMainProcessor.cxx:main:129]
2019-03-15T14:33:10.865996 D23515 maja-processing-3.2.2 3.2 [000000028370] [E] [vnsMajaMainProcessor.cxx:main:129]
And yes it's correct that there are no L1 products in this folder, but there shouldn't be. /media/au173212/bigdrive/JEM/Orkideer/temp_maja is the maja working directory. Hope you can help me?
Hello,
I downloaded the last version of maja3.3_TM and i need the appropriate gipp files, to get l2a product using sen2agri platform. I looked through this url http://tully.ups-tlse.fr/olivier/gipp_maja/tree/master, but i couldn't find the hole gipp files, like the gip tile id CMN000.
Hello,
I installed MAJA 3.3.0 TM with Start_maja on a CentOS 7.6 but the result I get (without errors) is of a much lower definition than what is downloadable from https://theia.cnes.fr/atdistrib/rocket/#/home.
One of the masks downloaded from theia's site (converted to jpg because I can not attach a tif)
The corresponding mask I generated
Can anyone tell me in which direction I should look ?
Thanks !
My configuration:
Commande is: "python ./start_maja.py -f folders.txt -g GIPP_MAJA_S2A_S2B_TM -l
LUT_MAJA_S2A_S2B_CAMS_H2ONew_20190411 -t 31UER -s Wallonie -d 20190506 -e
20190516"
I had problem while running MAJA 3.2.2, it gave the following error in the log file S2A_MSIL1C_20190211T111201_N0207_R137_T30STB_20190211T114747.SAFE.log
/opt/MAJA/bin/maja-processing-3.2.2: error while loading shared libraries: libMajaChain.so.3.2: cannot open shared object file: No such file or directory
We tried to resolve the problem by setting the environment variable LD_LIBRARY_PATH to /opt/MAJA/lib where the library libMajaChain.so.3.2 is stored.
I started the MAJA processing on all the dates in the time series and the first backward processing was unsuccessful.
Here's the error in the log:
/home/oo/Tools/MAJA/bin/maja-processing-3.3.2: error while loading shared libraries: libMajaChain.so.3.3: cannot open shared object file: No such file or directory
and here's the folders.txt configuration file:
repCode=/home/oo/Tools/Start-MAJA-master
repWork=/mnt/d_physiq/temp/MAJA
repL1 =/mnt/g_verbatim/MAJA/L1C_PDGS
repL2 =/mnt/d_physiq/MAJA/L2A_MAJA
repMaja=/home/oo/Tools/MAJA/bin/maja
repCAMS=/mnt/d_physiq/MAJA/CAMS
Hi!
If I start Maja including the backward initialization it works fine. If the processing is continued it crashes.
I think the reason is that L2type is overwritten when parsing the existing files, i.e. not specified for the processing later on.
line 382: L2type = Non
should be moved before the loop (e.g. to line 372)
Right?
The original Maja output:
python ./start_maja.py -f folders.txt -c GIPP -t 33UWP -s 33UWP -d 20180701 -e 20180707 --debug -v
2019-04-04 10:11:53,536 - Start-Maja - INFO - repCAMS is missing from configuration file. Needed : repCode, repWork, repL1, repL2, repMaja
2019-04-04 10:11:53,536 - Start-Maja - INFO - Processing without CAMS
2019-04-04 10:11:53,560 - Start-Maja - INFO - Most recent processed date : 20180705
2019-04-04 10:11:53,560 - Start-Maja - INFO - => processing date 20180707
Traceback (most recent call last):
File "./start_maja.py", line 637, in
start_maja(folder_file, context, site, tile, orbit, nb_backward, options, debug_mode)
File "./start_maja.py", line 482, in start_maja
logger.info("MAJA command failed : %s", commande)
UnboundLocalError: local variable 'commande' referenced before assignment
I put in place MAJA and I got an error during writing of HDR CAMS data.
I run with python 3.7, I modified the script with this solution (add .decode("utf-8") instead of encoding="UTF-8"))
ERROR :
line 281, in process_one_file
f.write(ET.tostring(tree, pretty_print=True, xml_declaration=True,encoding="UTF-8"))
TypeError: write() argument must be str, not bytes
Hello!
I want to do cloud masking of L2A product by using maja. So I dont need to use L1C product, but when I work the program, it tells me error which is "Start-Maja - ERROR - No L1C product found in ...(directory)". Is it possible to ignore that?
when i run DTMCreation with Avignon example, i have error :
gdalwarp: error while loading shared libraries: libgdal.so.1: cannot open shared object file: No such file or directory
Traceback (most recent call last):
File "DTMCreation.py", line 326, in
creator.run(args.out, args.tempout)
File "DTMCreation.py", line 286, in run
water_zipped = self.WaterZipped)
File "/home/pascal/Start-MAJA/prepare_dtm/../prepare_dtm/tuilage_mnt_eau_S2.py", line 232, in run
mnt_90m.decoupe(fic_mnt_in)
File "/home/pascal/Start-MAJA/prepare_dtm/../prepare_dtm/lib_mnt.py", line 325, in decoupe
(nblig, nbcol, type_donnee, endian) = lire_entete_mnt(fic_hdr_mnt)
File "/home/pascal/Start-MAJA/prepare_dtm/../prepare_dtm/lib_mnt.py", line 193, in lire_entete_mnt
with open(fic_hdr, 'r') as f:
IOError: [Errno 2] No such file or directory: '/tmp/31TFJyuadfb/tmpg1573N_90m.hdr'
hi,
i tested maja on sentinel products, and i'm wondering if does maja work on landsat8 products.
Hello,
I would like to know if you integrated Landsat 8 data on the updated version of start-maja.
Hi,
I have a new error with this script, it seems to be the TILE_ID in MTD_TL.xml file.
Here are two examples of TILE_ID from different xml files :
<TILE_ID metadataLevel="Brief">S2A_OPER_MSI_L1C_TL_SGS__20180228T014607_A014027_T54HYF_N02.06</TILE_ID>
<TILE_ID metadataLevel="Brief">S2A_OPER_MSI_L1C_TL_EPAE_20190424T015912_A020033_T54HYF_N02.07</TILE_ID>
The archiving centre is not the same (SGS and EPA), and when you split this string, the lenght of the list is different because of underscore number (11 & 10).
Here is the error :
python DTMCreation.py -i /media/tmp/S2A_MSIL1C_20190424T002711_N0207_R016_T54HYF_20190424T015912.SAFE/ -s /media/raster/monde/SRTM/ -w /media/raster/monde/SWBD/ -o /media/raster/MAJA/DTM/S2/54HYF/
Found SRTM zip-archives...
Found Water zip-archives...
Traceback (most recent call last):
File "DTMCreation.py", line 326, in
creator.run(args.out, args.tempout)
File "DTMCreation.py", line 274, in run
self.site = self.getSiteInfo(self.mtd, self.dtype)
File "DTMCreation.py", line 210, in getSiteInfo
assert len(tileFilenameItems) == 11
AssertionError
Hi,
We are planning a large project of processing Sentinel-2 covering the whole Norway, but we noticed that the current version of Maja does not support processing data beyond 60 degrees north.
Is it true that Maja V3.3 can only process data within the SRTM range or is it possible to tweak the SRTM DTM and the SBWD to one's own dataset that is beyond the SRTM range? We really want to process data beyond 60 degrees north, but this is outside the SRTM range. We have a free and open DTM that we can use.
My following-up question is, will the next version of Maja be possible to use your own DEM or other DEMs that goes above the SRTM range? And if so, do you by chance know when this new version of Maja will be released?
Many thanks for your help!
Kind regards,
Dear Olivier,
I'm trying to process L1C images from peps but I have the error below :
PreProcessing: vns::Data::ERROR: DataApplicationHandler(0x1ec97f0): In the input directory, there is no GIPP input data matching with the 'AUX_REFDE2' File_Type pattern! [vnsDataApplicationHandler.cxx:1699] [MAJA Data Exception] [vnsMajaMainProcessor.cxx:main:129]
May we miss something during the setup of MAJA ?
Thank you very much !
Antoine Lefebvre
Hello Olivier,
I am running MAJA-3.3.0-TM for 6 scenes:
S2B_MSIL1C_20190713T173909_N0208_R098_T14UNU_20190713T211057
S2A_MSIL1C_20181207T173711_N0207_R055_T14UNU_20181207T192326
S2B_MSIL1C_20190223T174329_N0207_R098_T14UNU_20190223T211941
S2A_MSIL1C_20181117T173611_N0207_R055_T14UNU_20181117T210544
S2A_MSIL1C_20190509T173911_N0207_R098_T14UNU_20190509T224452
S2A_MSIL1C_20190320T174031_N0207_R098_T14UNU_20190320T230030
I got the following error: The number of NoData pixel in the output L2 composite product is too high.
I have checked the log file (attached. It was ran with --debug mode) and found that the L2outNoDataRate (pourcentage) for one of the scenes is higher than the MaxNoDataPercentage allow, so I guess that's why it failed. However, from the log file I cannot figure out which one is the image.
Also, I am wondering why it fails here instead of creating a folder such as for the cloud cover when it doesn't fit the requirements and if there is a way of avoiding it.
Thank you very much
Ruben
S2A_MSIL1C_20181117T173611_N0207_R055_T14UNU_20181117T210544.SAFE.log
Platform and versions:
• Ubuntu 18.04.4 LTS
• Python 3.7.6 (default, Jan 8 2020, 19:59:22)
[GCC 7.3.0] :: Anaconda, Inc. on linux
• GDAL 2.4.2, released 2019/06/28
Dear Maja Users,
I have some problems with the folder and files structure required to properly run Maja. To start, I want to first process without CAMS. The steps are described below:
1-Downloaded and installed Maja_3.3.2_TM.run
bash MAJA-3.2.2_TM.run –target ~/maja
2.Clone the current repository to get start_maja.py
git clone https://github.com/CNES/Start-MAJA.git
3. Prepare folders.txt
repCode= ~/maja/Start-MAJA
repWork= /media/ernesto/cuatrot/maja_procesos/temporal
repL1 = /media/ernesto/cuatrot/maja_procesos/S2/L1C/L1CCH
repL2 = /media/ernesto/cuatrot/maja_procesos/S2/L2A/L2ACH
repMaja= ~/maja/bin/maja
repCAMS= /media/ernesto/cuatrot/maja_procesos/cams
4. Retrieve Sentinel-2 L1C data
Using peps_download.py, I have downloaded 188 images for 4 tiles (PDU,PDV,PEU,PEV) for the period from 2019-08-01 to 2020-03-20 . An example of the commands used to download images for one tile is presented:
python /media/ernesto/cuatrot/peps_download/peps_download-master/peps_download.py -c S2ST -t 16PDU -o 140 -a /media/ernesto/cuatrot/peps_download/peps_download-master/peps.txt -d 2019-08-01 -f 2020-03-20
Unzip L1C data path=media/ernesto/cuatrot/maja_procesos/S2/L1C/L1CCH
5. Parameters
path=~/maja/Start-MAJA/userconf
The files in this folder are:
MAJAUserConfig_FORMOSAT_MUSCATE_PROTO.xml
MAJAUserConfig_LANDSAT8_MUSCATE_PROTO.xml
MAJAUserConfig_LANDSAT8_MUSCATE.xml
MAJAUserConfig_LANDSAT8.xml
MAJAUserConfig_LANDSAT_MUSCATE_PROTO.xml
MAJAUserConfig_LANDSAT_MUSCATE.xml
MAJAUserConfig_SENTINEL2_GPP.xml
MAJAUserConfig_SENTINEL2_MUSCATE.xml
MAJAUserConfig_SENTINEL2_TM.xml
MAJAUserConfig_SENTINEL2.xml
MAJAUserConfig_SPOT4_MUSCATE_PROTO.xml
MAJAUserConfigSystem.xml
MAJAUserConfig_VENUS.xml
path=~/maja/Start-MAJA/GIPP/SENTINEL2_TM
The files in this folder are:
README.md
S2A_TEST_GIP_CKEXTL_S_ALLSITES_00001_20190626_21000101.EEF
S2A_TEST_GIP_CKEXTL_S_ALLSITES_10001_20190626_21000101.EEF
S2A_TEST_GIP_CKQLTL_S_ALLSITES_00001_20190626_21000101.EEF
S2A_TEST_GIP_CKQLTL_S_ALLSITES_10001_20190626_21000101.EEF
S2A_TEST_GIP_L2COMM_L_ALLSITES_00001_20190626_21000101.EEF
S2A_TEST_GIP_L2COMM_L_ALLSITES_10001_20190626_21000101.EEF
S2A_TEST_GIP_L2SMAC_L_ALLSITES_00001_20190626_21000101.EEF
S2B_TEST_GIP_CKEXTL_S_ALLSITES_00001_20190626_21000101.EEF
S2B_TEST_GIP_CKEXTL_S_ALLSITES_10001_20190626_21000101.EEF
S2B_TEST_GIP_CKQLTL_S_ALLSITES_00001_20190626_21000101.EEF
S2B_TEST_GIP_CKQLTL_S_ALLSITES_10001_20190626_21000101.EEF
S2B_TEST_GIP_L2COMM_L_ALLSITES_00001_20190626_21000101.EEF
S2B_TEST_GIP_L2COMM_L_ALLSITES_10001_20190626_21000101.EEF
S2B_TEST_GIP_L2SMAC_L_ALLSITES_00001_20190626_21000101.EEF
S2__TEST_GIP_L2SITE_S_ALLSITES_00001_20190626_21000101.EEF
I have a doubt regarding the use of GIPP without CAMS. According to the instructions, if we want to use Parameters without CAMS, we should use the folder SENTINEL2_TM_CAMS:
However, when we read the README.md for http://tully.ups-tlse.fr/olivier/gipp_maja/tree/master/SENTINEL2_TM_CAMS. It clearly states that CAMS option enabled. And the README.md for: http://tully.ups-tlse.fr/olivier/gipp_maja/tree/master/SENTINEL2_TM, we get the message that CAMS option disabled.
So, I am confused. I assumed that the README files are correct and I should use SENTINEL2_TM when processing without CAMS.
I downloaded the LUT from the zenodo server and I choose to use: 20190626_LUT_MAJA_SENTINEL2_TM_NOCAMS
path=~/maja/Start-MAJA/20190626_LUT_MAJA_SENTINEL2_TM_NOCAMS/LUTs
Files in this folder:
S2A_TEST_GIP_L2ALBD_L_CONTINEN_00001_20190626_21000101.DBL.DIR
S2A_TEST_GIP_L2ALBD_L_CONTINEN_00001_20190626_21000101.HDR
S2A_TEST_GIP_L2DIFT_L_CONTINEN_00001_20190626_21000101.DBL.DIR
S2A_TEST_GIP_L2DIFT_L_CONTINEN_00001_20190626_21000101.HDR
S2A_TEST_GIP_L2DIRT_L_CONTINEN_00001_20190626_21000101.DBL.DIR
S2A_TEST_GIP_L2DIRT_L_CONTINEN_00001_20190626_21000101.HDR
S2A_TEST_GIP_L2TOCR_L_CONTINEN_00001_20190626_21000101.DBL.DIR
S2A_TEST_GIP_L2TOCR_L_CONTINEN_00001_20190626_21000101.HDR
S2A_TEST_GIP_L2WATV_L_CONTINEN_00001_20190626_21000101.DBL.DIR
S2A_TEST_GIP_L2WATV_L_CONTINEN_00001_20190626_21000101.HDR
S2B_TEST_GIP_L2ALBD_L_CONTINEN_00001_20190626_21000101.DBL.DIR
S2B_TEST_GIP_L2ALBD_L_CONTINEN_00001_20190626_21000101.HDR
S2B_TEST_GIP_L2DIFT_L_CONTINEN_00001_20190626_21000101.DBL.DIR
S2B_TEST_GIP_L2DIFT_L_CONTINEN_00001_20190626_21000101.HDR
S2B_TEST_GIP_L2DIRT_L_CONTINEN_00001_20190626_21000101.DBL.DIR
S2B_TEST_GIP_L2DIRT_L_CONTINEN_00001_20190626_21000101.HDR
S2B_TEST_GIP_L2TOCR_L_CONTINEN_00001_20190626_21000101.DBL.DIR
S2B_TEST_GIP_L2TOCR_L_CONTINEN_00001_20190626_21000101.HDR
S2B_TEST_GIP_L2WATV_L_CONTINEN_00001_20190626_21000101.DBL.DIR
S2B_TEST_GIP_L2WATV_L_CONTINEN_00001_20190626_21000101.HDR
6. DTM preparation
After some problems with the code to prepare the dtm. It was suggested to use another branch of the DTM preparation code (https://github.com/CNES/Start-MAJA/tree/reprog-rc1/prepare_mnt) and using this updated code it works fine.
path=~/maja/Start-MAJA/DTM
Since I use 4 tiles I produced 4 DTMs. This folder contains 2 files per tile:
S2__TEST_AUX_REFDE2_16PDU_0001.DBL.DIR S2__TEST_AUX_REFDE2_16PEU_0001.DBL.DIR
S2__TEST_AUX_REFDE2_16PDU_0001.HDR S2__TEST_AUX_REFDE2_16PEU_0001.HDR
S2__TEST_AUX_REFDE2_16PDV_0001.DBL.DIR S2__TEST_AUX_REFDE2_16PEV_0001.DBL.DIR
S2__TEST_AUX_REFDE2_16PDV_0001.HDR S2__TEST_AUX_REFDE2_16PEV_0001.HDR
7. CAMS data
As stated in the beginning currently I want to use MAJA without CAMS but I would like to be able to process with CAMS in the future.
Following instructions from https://github.com/CNES/Start-MAJA/tree/master/cams_download
At this moment I have registered at ECMWF and I have API key.
I do not know what it means with: Create the file '.ecmwfapirc' in your home and type the API key. How you do that. Is it a text in which I hould put the API key? And referring to home folder you mean Home (~) or it means the “cams_download” folder in the Statr-Maja folder?
path= ~/maja/Start-MAJA/cams_download
8. Execute start_maja.py
In the start-maja folder (~/maja/Start-MAJA), I open terminal and run:
python start_maja.py -f ~/maja/Start-MAJA/folders.txt -g ~/maja/Start-MAJA/GIPP/SENTINEL2_TM -l ~/maja/Start-MAJA/20190626_LUT_MAJA_SENTINEL2_TM_NOCAMS -t 16PDU -s ser -d 20190801 -e 202000320
Running the above commands produce the following errors:
What am I missing or doing wrong? Many thanks for your help.
Kind regards,
Hello and thanks for the great work on this algorithm. I'm interested in adapting MAJA's time series based method for cloud detection to Planet Labs imagery and was curious if there are plans to make the source code available?
Hi,
Recently installed and trying to test MAJA but encountering error processing some dates.
My setup:
OS = Ubuntu 18.04
CPU = 8 cores
RAM = 12 GB
maja = 3.3.0 TM
start_maja = latest master
LUT = LUT_MAJA_S2A_S2B_CAMS_H2ONew_20190411.tgz
S2L1C = tile 30UYC with date range 20180601 to 20180701 and 90% cloud threshold
python /data/3.3/git/maja/start_maja.py -f /data/3.3/git/maja/folders.txt -g GIPP_S2_MAJA_3.3_TM -l LUT_MAJA_3_TM_CAMS -t 30UYC -s t30UYC -d 20180601 -e 20180701 -z
Error message (after attempting to re-run):
2019-05-21 09:27:36,828 - Start-Maja - INFO - Most recent processed date : 20180606
2019-05-21 09:27:36,851 - Start-Maja - INFO - => processing date 20180608
2019-05-21 09:27:38,017 - Start-Maja - INFO - Using MUSCATE L2 type
2019-05-21 09:27:38,023 - Start-Maja - INFO - previous L2 : /data/SENTINEL2/L2A_maja/t30UYC/30UYC/GIPP_S2_MAJA_3.3_TM/SENTINEL2A_20180606-111256-734_L2A_T30UYC_C_V1-0
2019-05-21 09:28:09,639 - Start-Maja - INFO - #################################
2019-05-21 09:28:09,639 - Start-Maja - INFO - #################################
2019-05-21 09:28:09,639 - Start-Maja - INFO - processing /data/SENTINEL2/L1C/t30UYC/S2B_MSIL1C_20180608T105649_N0206_R094_T30UYC_20180608T120643.zip in nominal mode
2019-05-21 09:28:09,639 - Start-Maja - INFO - MAJA logfile: /data/SENTINEL2/L2A_maja/t30UYC/30UYC/GIPP_S2_MAJA_3.3_TM//S2B_MSIL1C_20180608T105649_N0206_R094_T30UYC_20180608T120643.zip.log
2019-05-21 09:28:09,639 - Start-Maja - INFO - #################################
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
L2A product /data/SENTINEL2/L2A_maja/t30UYC/30UYC/GIPP_S2_MAJA_3.3_TM/SENTINEL2B_20180608-110550-630_L2A_T30UYC_C_V1-0 not found
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
2019-05-21 09:41:48,853 - Start-Maja - INFO - => processing date 20180611
2019-05-21 09:41:49,414 - Start-Maja - INFO - Using MUSCATE L2 type
2019-05-21 09:41:49,423 - Start-Maja - INFO - previous L2 : /data/SENTINEL2/L2A_maja/t30UYC/30UYC/GIPP_S2_MAJA_3.3_TM/SENTINEL2B_20180608-110550-630_L2A_T30UYC_C_V1-0
2019-05-21 09:42:23,512 - Start-Maja - INFO - #################################
2019-05-21 09:42:23,512 - Start-Maja - INFO - #################################
2019-05-21 09:42:23,512 - Start-Maja - INFO - processing /data/SENTINEL2/L1C/t30UYC/S2B_MSIL1C_20180611T110619_N0206_R137_T30UYC_20180611T170311.zip in nominal mode
2019-05-21 09:42:23,512 - Start-Maja - INFO - MAJA logfile: /data/SENTINEL2/L2A_maja/t30UYC/30UYC/GIPP_S2_MAJA_3.3_TM//S2B_MSIL1C_20180611T110619_N0206_R137_T30UYC_20180611T170311.zip.log
2019-05-21 09:42:23,512 - Start-Maja - INFO - #################################
2019-05-21T09:54:47.369499 proc-06-03 maja-processing-3.3.0 3.3 [000000061661] [E] vns::Plugin::ERROR: Sentinel2TML2ImageFileReader(0x86dee0): The MUSCATE product </data/SENTINEL2/MAJA/t30UYC/30UYC/GIPP_S2_MAJA_3.3_TM//in/SENTINEL2B_20180608-110550-630_L2A_T30UYC_C_V1-0/SENTINEL2B_20180608-110550-630_L2A_T30UYC_C_V1-0_MTD_ALL.xml> is not valid according to JPI! [vnsMuscateL2ImageFileReader.txx:MuscateDetectL2Products:368]
2019-05-21T09:54:47.408363 proc-06-03 maja-processing-3.3.0 3.3 [000000061661] [E] ****************************************************************************************************** [vnsMajaMainProcessor.cxx:main:130]
2019-05-21T09:54:47.412682 proc-06-03 maja-processing-3.3.0 3.3 [000000061661] [E] ScientificProcessing: vns::Plugin::ERROR: Sentinel2TML2ImageFileReader(0x86dee0): The MUSCATE product </data/SENTINEL2/MAJA/t30UYC/30UYC/GIPP_S2_MAJA_3.3_TM//in/SENTINEL2B_20180608-110550-630_L2A_T30UYC_C_V1-0/SENTINEL2B_20180608-110550-630_L2A_T30UYC_C_V1-0_MTD_ALL.xml> is not valid according to JPI! [vnsMuscateL2ImageFileReader.txx:368] [MAJA Base Exception] [vnsMajaMainProcessor.cxx:main:130]
2019-05-21T09:54:47.416174 proc-06-03 maja-processing-3.3.0 3.3 [000000061661] [E] [vnsMajaMainProcessor.cxx:main:130]
2019-05-21T09:54:47.419437 proc-06-03 maja-processing-3.3.0 3.3 [000000061661] [E] ****************************************************************************************************** [vnsMajaMainProcessor.cxx:main:130]
2019-05-21 09:54:47,462 - Start-Maja - INFO - #######################################
2019-05-21 09:54:47,462 - Start-Maja - INFO - Error detected, see: /data/SENTINEL2/L2A_maja/t30UYC/30UYC/GIPP_S2_MAJA_3.3_TM//S2B_MSIL1C_20180611T110619_N0206_R137_T30UYC_20180611T170311.zip.log
2019-05-21 09:54:47,462 - Start-Maja - INFO - #######################################
I have looked at the raw S2L1C images (20180606, 20180608 and 20180611) and they all look fine. Any ideas whats going wrong?
Thanks
I am trying to automate the SRTM retrieval step using https://github.com/cmla/srtm4 and using these download SRTM files to determine the SBWD data aswell, as I understand it this can actually be retrieved from the nodata values in the SRTM file as these are water. Is this correct? And if this should be possible, could you give me some guidance on how to implement this?
I tried to make a test run of start-maja on a few Sentinel-2 tiles for Hamburg. I use Python 2.7 and CentOS 7. When I execute start_maja.py with the command
python ./start_maja.py -f myfolders.txt -g ../gipp/SENTINEL2_NATIF -l ../luts/LUT_MAJA_3_TM_CAMS -t 32UNE -s Hamburg -d 20190714 -e 20190717
the script start fine. I also managed to create a DTM file for Hamburg before the test run. After a few seconds I obtain the following error message:
2020-04-02T09:02:40.829532 janwevers.uservm.rscloud.vito.be maja-processing-3.3.5 3.3 [000000018205] [E] vns::Data::ERROR: DataApplicationHandler(0x2006a00): DataApplicationHandler::GetGippFilename: More than one input filename have been found for the File_Type 'GIP_L2WATV' and for the satellite 'SENTINEL-2B'.
See details further below. Many thanks and best wishes, Ralf.
The files in ../proc/tmp/maja/Hamburg/gipp/SENTINEL2_NATIF/in are
S2A_TEST_GIP_CKEXTL_S_ALLSITES_00001_20190626_21000101.EEF
S2A_TEST_GIP_CKQLTL_S_ALLSITES_00001_20190626_21000101.EEF
S2A_TEST_GIP_L2ALBD_L_BLACKCAR_50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2ALBD_L_BLACKCAR_50001_00000000_99999999.HDR
S2A_TEST_GIP_L2ALBD_L_CONTINEN_00001_20190626_21000101.DBL.DIR
S2A_TEST_GIP_L2ALBD_L_CONTINEN_00001_20190626_21000101.HDR
S2A_TEST_GIP_L2ALBD_L_CONTINEN_10005_20150703_21000101.DBL.DIR
S2A_TEST_GIP_L2ALBD_L_CONTINEN_10005_20150703_21000101.HDR
S2A_TEST_GIP_L2ALBD_L_DUST_____50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2ALBD_L_DUST_____50001_00000000_99999999.HDR
S2A_TEST_GIP_L2ALBD_L_ORGANICM_50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2ALBD_L_ORGANICM_50001_00000000_99999999.HDR
S2A_TEST_GIP_L2ALBD_L_SEASALT__50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2ALBD_L_SEASALT__50001_00000000_99999999.HDR
S2A_TEST_GIP_L2ALBD_L_SULPHATE_50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2ALBD_L_SULPHATE_50001_00000000_99999999.HDR
S2A_TEST_GIP_L2COMM_L_ALLSITES_00001_20190626_21000101.EEF
S2A_TEST_GIP_L2DIFT_L_BLACKCAR_50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2DIFT_L_BLACKCAR_50001_00000000_99999999.HDR
S2A_TEST_GIP_L2DIFT_L_CONTINEN_00001_20190626_21000101.DBL.DIR
S2A_TEST_GIP_L2DIFT_L_CONTINEN_00001_20190626_21000101.HDR
S2A_TEST_GIP_L2DIFT_L_CONTINEN_10005_20150703_21000101.DBL.DIR
S2A_TEST_GIP_L2DIFT_L_CONTINEN_10005_20150703_21000101.HDR
S2A_TEST_GIP_L2DIFT_L_DUST_____50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2DIFT_L_DUST_____50001_00000000_99999999.HDR
S2A_TEST_GIP_L2DIFT_L_ORGANICM_50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2DIFT_L_ORGANICM_50001_00000000_99999999.HDR
S2A_TEST_GIP_L2DIFT_L_SEASALT__50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2DIFT_L_SEASALT__50001_00000000_99999999.HDR
S2A_TEST_GIP_L2DIFT_L_SULPHATE_50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2DIFT_L_SULPHATE_50001_00000000_99999999.HDR
S2A_TEST_GIP_L2DIRT_L_BLACKCAR_50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2DIRT_L_BLACKCAR_50001_00000000_99999999.HDR
S2A_TEST_GIP_L2DIRT_L_CONTINEN_00001_20190626_21000101.DBL.DIR
S2A_TEST_GIP_L2DIRT_L_CONTINEN_00001_20190626_21000101.HDR
S2A_TEST_GIP_L2DIRT_L_CONTINEN_10005_20150703_21000101.DBL.DIR
S2A_TEST_GIP_L2DIRT_L_CONTINEN_10005_20150703_21000101.HDR
S2A_TEST_GIP_L2DIRT_L_DUST_____50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2DIRT_L_DUST_____50001_00000000_99999999.HDR
S2A_TEST_GIP_L2DIRT_L_ORGANICM_50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2DIRT_L_ORGANICM_50001_00000000_99999999.HDR
S2A_TEST_GIP_L2DIRT_L_SEASALT__50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2DIRT_L_SEASALT__50001_00000000_99999999.HDR
S2A_TEST_GIP_L2DIRT_L_SULPHATE_50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2DIRT_L_SULPHATE_50001_00000000_99999999.HDR
S2A_TEST_GIP_L2SMAC_L_ALLSITES_00001_20190626_21000101.EEF
S2A_TEST_GIP_L2TOCR_L_BLACKCAR_50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2TOCR_L_BLACKCAR_50001_00000000_99999999.HDR
S2A_TEST_GIP_L2TOCR_L_CONTINEN_00001_20190626_21000101.DBL.DIR
S2A_TEST_GIP_L2TOCR_L_CONTINEN_00001_20190626_21000101.HDR
S2A_TEST_GIP_L2TOCR_L_CONTINEN_10005_20150703_21000101.DBL.DIR
S2A_TEST_GIP_L2TOCR_L_CONTINEN_10005_20150703_21000101.HDR
S2A_TEST_GIP_L2TOCR_L_DUST_____50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2TOCR_L_DUST_____50001_00000000_99999999.HDR
S2A_TEST_GIP_L2TOCR_L_ORGANICM_50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2TOCR_L_ORGANICM_50001_00000000_99999999.HDR
S2A_TEST_GIP_L2TOCR_L_SEASALT__50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2TOCR_L_SEASALT__50001_00000000_99999999.HDR
S2A_TEST_GIP_L2TOCR_L_SULPHATE_50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2TOCR_L_SULPHATE_50001_00000000_99999999.HDR
S2A_TEST_GIP_L2WATV_L_CONTINEN_00001_20190626_21000101.DBL.DIR
S2A_TEST_GIP_L2WATV_L_CONTINEN_00001_20190626_21000101.HDR
S2A_TEST_GIP_L2WATV_L_CONTINEN_10006_20150703_21000101.DBL.DIR
S2A_TEST_GIP_L2WATV_L_CONTINEN_10006_20150703_21000101.HDR
S2B_MSIL1C_20190714T103029_N0208_R108_T32UNE_20190714T122554.SAFE
S2B_MSIL1C_20190717T104029_N0208_R008_T32UNE_20190717T142616.SAFE
S2B_TEST_GIP_CKEXTL_S_ALLSITES_00001_20190626_21000101.EEF
S2B_TEST_GIP_CKQLTL_S_ALLSITES_00001_20190626_21000101.EEF
S2B_TEST_GIP_L2ALBD_L_BLACKCAR_50001_00000000_99999999.DBL.DIR
S2B_TEST_GIP_L2ALBD_L_BLACKCAR_50001_00000000_99999999.HDR
S2B_TEST_GIP_L2ALBD_L_CONTINEN_00001_20190626_21000101.DBL.DIR
S2B_TEST_GIP_L2ALBD_L_CONTINEN_00001_20190626_21000101.HDR
S2B_TEST_GIP_L2ALBD_L_CONTINEN_10003_20150703_21000101.DBL.DIR
S2B_TEST_GIP_L2ALBD_L_CONTINEN_10003_20150703_21000101.HDR
S2B_TEST_GIP_L2ALBD_L_DUST_____50001_00000000_99999999.DBL.DIR
S2B_TEST_GIP_L2ALBD_L_DUST_____50001_00000000_99999999.HDR
S2B_TEST_GIP_L2ALBD_L_ORGANICM_50001_00000000_99999999.DBL.DIR
S2B_TEST_GIP_L2ALBD_L_ORGANICM_50001_00000000_99999999.HDR
S2B_TEST_GIP_L2ALBD_L_SEASALT__50001_00000000_99999999.DBL.DIR
S2B_TEST_GIP_L2ALBD_L_SEASALT__50001_00000000_99999999.HDR
S2B_TEST_GIP_L2ALBD_L_SULPHATE_50001_00000000_99999999.DBL.DIR
S2B_TEST_GIP_L2ALBD_L_SULPHATE_50001_00000000_99999999.HDR
S2B_TEST_GIP_L2COMM_L_ALLSITES_00001_20190626_21000101.EEF
S2B_TEST_GIP_L2DIFT_L_BLACKCAR_50001_00000000_99999999.DBL.DIR
S2B_TEST_GIP_L2DIFT_L_BLACKCAR_50001_00000000_99999999.HDR
S2B_TEST_GIP_L2DIFT_L_CONTINEN_00001_20190626_21000101.DBL.DIR
S2B_TEST_GIP_L2DIFT_L_CONTINEN_00001_20190626_21000101.HDR
S2B_TEST_GIP_L2DIFT_L_CONTINEN_10002_20150703_21000101.DBL.DIR
S2B_TEST_GIP_L2DIFT_L_CONTINEN_10002_20150703_21000101.HDR
S2B_TEST_GIP_L2DIFT_L_DUST_____50001_00000000_99999999.DBL.DIR
S2B_TEST_GIP_L2DIFT_L_DUST_____50001_00000000_99999999.HDR
S2B_TEST_GIP_L2DIFT_L_ORGANICM_50001_00000000_99999999.DBL.DIR
S2B_TEST_GIP_L2DIFT_L_ORGANICM_50001_00000000_99999999.HDR
S2B_TEST_GIP_L2DIFT_L_SEASALT__50001_00000000_99999999.DBL.DIR
S2B_TEST_GIP_L2DIFT_L_SEASALT__50001_00000000_99999999.HDR
S2B_TEST_GIP_L2DIFT_L_SULPHATE_50001_00000000_99999999.DBL.DIR
S2B_TEST_GIP_L2DIFT_L_SULPHATE_50001_00000000_99999999.HDR
S2B_TEST_GIP_L2DIRT_L_BLACKCAR_50001_00000000_99999999.DBL.DIR
S2B_TEST_GIP_L2DIRT_L_BLACKCAR_50001_00000000_99999999.HDR
S2B_TEST_GIP_L2DIRT_L_CONTINEN_00001_20190626_21000101.DBL.DIR
S2B_TEST_GIP_L2DIRT_L_CONTINEN_00001_20190626_21000101.HDR
S2B_TEST_GIP_L2DIRT_L_CONTINEN_10002_20150703_21000101.DBL.DIR
S2B_TEST_GIP_L2DIRT_L_CONTINEN_10002_20150703_21000101.HDR
S2B_TEST_GIP_L2DIRT_L_DUST_____50001_00000000_99999999.DBL.DIR
S2B_TEST_GIP_L2DIRT_L_DUST_____50001_00000000_99999999.HDR
S2B_TEST_GIP_L2DIRT_L_ORGANICM_50001_00000000_99999999.DBL.DIR
S2B_TEST_GIP_L2DIRT_L_ORGANICM_50001_00000000_99999999.HDR
S2B_TEST_GIP_L2DIRT_L_SEASALT__50001_00000000_99999999.DBL.DIR
S2B_TEST_GIP_L2DIRT_L_SEASALT__50001_00000000_99999999.HDR
S2B_TEST_GIP_L2DIRT_L_SULPHATE_50001_00000000_99999999.DBL.DIR
S2B_TEST_GIP_L2DIRT_L_SULPHATE_50001_00000000_99999999.HDR
S2B_TEST_GIP_L2SMAC_L_ALLSITES_00001_20190626_21000101.EEF
S2B_TEST_GIP_L2TOCR_L_BLACKCAR_50001_00000000_99999999.DBL.DIR
S2B_TEST_GIP_L2TOCR_L_BLACKCAR_50001_00000000_99999999.HDR
S2B_TEST_GIP_L2TOCR_L_CONTINEN_00001_20190626_21000101.DBL.DIR
S2B_TEST_GIP_L2TOCR_L_CONTINEN_00001_20190626_21000101.HDR
S2B_TEST_GIP_L2TOCR_L_CONTINEN_10002_20150703_21000101.DBL.DIR
S2B_TEST_GIP_L2TOCR_L_CONTINEN_10002_20150703_21000101.HDR
S2B_TEST_GIP_L2TOCR_L_DUST_____50001_00000000_99999999.DBL.DIR
S2B_TEST_GIP_L2TOCR_L_DUST_____50001_00000000_99999999.HDR
S2B_TEST_GIP_L2TOCR_L_ORGANICM_50001_00000000_99999999.DBL.DIR
S2B_TEST_GIP_L2TOCR_L_ORGANICM_50001_00000000_99999999.HDR
S2B_TEST_GIP_L2TOCR_L_SEASALT__50001_00000000_99999999.DBL.DIR
S2B_TEST_GIP_L2TOCR_L_SEASALT__50001_00000000_99999999.HDR
S2B_TEST_GIP_L2TOCR_L_SULPHATE_50001_00000000_99999999.DBL.DIR
S2B_TEST_GIP_L2TOCR_L_SULPHATE_50001_00000000_99999999.HDR
S2B_TEST_GIP_L2WATV_L_CONTINEN_00001_20190626_21000101.DBL.DIR
S2B_TEST_GIP_L2WATV_L_CONTINEN_00001_20190626_21000101.HDR
S2B_TEST_GIP_L2WATV_L_CONTINEN_10006_20150703_21000101.DBL.DIR
S2B_TEST_GIP_L2WATV_L_CONTINEN_10006_20150703_21000101.HDR
S2__TEST_AUX_REFDE2_T32UNE_0001.DBL.DIR
S2__TEST_AUX_REFDE2_T32UNE_0001.HDR
S2__TEST_GIP_L2SITE_S_ALLSITES_00001_20190626_21000101.EEF
The complete log output is:
2020-04-02 09:02:22,368 - Start-Maja - INFO - repCAMS is missing from configuration file. Needed : repCode, repWork, repL1, repL2, repMaja
2020-04-02 09:02:22,368 - Start-Maja - INFO - Processing without CAMS
2020-04-02 09:02:22,587 - Start-Maja - INFO - /home/janwevers/worldcover/proc/out/maja/Hamburg/32UNE/../gipp/SENTINEL2_NATIF//S2?_OPER_SSC_L2VALD_32UNE____T20190714.DBL.DIR not found
2020-04-02 09:02:22,587 - Start-Maja - INFO - /home/janwevers/worldcover/proc/out/maja/Hamburg/32UNE/../gipp/SENTINEL2_NATIF//SENTINEL2?_20190714-*_T32UNE_C_V* not found
2020-04-02 09:02:22,590 - Start-Maja - INFO - /home/janwevers/worldcover/proc/out/maja/Hamburg/32UNE/../gipp/SENTINEL2_NATIF//S2?_OPER_SSC_L2VALD_32UNE____T20190717.DBL.DIR not found
2020-04-02 09:02:22,590 - Start-Maja - INFO - /home/janwevers/worldcover/proc/out/maja/Hamburg/32UNE/../gipp/SENTINEL2_NATIF//SENTINEL2?_20190717-*_T32UNE_C_V* not found
2020-04-02 09:02:22,590 - Start-Maja - INFO - No existing L2 product, we start with backward mode
2020-04-02 09:02:22,611 - Start-Maja - INFO - => processing date 20190714
2020-04-02 09:02:22,614 - Start-Maja - INFO - dates to process in backward mode :
2020-04-02 09:02:22,614 - Start-Maja - INFO - -- 20190714 : /home/janwevers/worldcover/proc/l1c/Hamburg/S2B_MSIL1C_20190714T103029_N0208_R108_T32UNE_20190714T122554.SAFE
2020-04-02 09:02:22,617 - Start-Maja - INFO - -- 20190717 : /home/janwevers/worldcover/proc/l1c/Hamburg/S2B_MSIL1C_20190717T104029_N0208_R008_T32UNE_20190717T142616.SAFE
2020-04-02 09:02:23,108 - Start-Maja - INFO - #################################
2020-04-02 09:02:23,108 - Start-Maja - INFO - #################################
2020-04-02 09:02:23,108 - Start-Maja - INFO - processing /home/janwevers/worldcover/proc/l1c/Hamburg/S2B_MSIL1C_20190714T103029_N0208_R108_T32UNE_20190714T122554.SAFE in backward mode
2020-04-02 09:02:23,109 - Start-Maja - INFO - Initialisation mode with backward is longer
2020-04-02 09:02:23,109 - Start-Maja - INFO - MAJA logfile: /home/janwevers/worldcover/proc/out/maja/Hamburg/32UNE/../gipp/SENTINEL2_NATIF//S2B_MSIL1C_20190714T103029_N0208_R108_T32UNE_20190714T122554.SAFE.log
2020-04-02 09:02:23,109 - Start-Maja - INFO - #################################
2020-04-02T09:02:40.829532 janwevers.uservm.rscloud.vito.be maja-processing-3.3.5 3.3 [000000018205] [E] vns::Data::ERROR: DataApplicationHandler(0x2006a00): DataApplicationHandler::GetGippFilename: More than one input filename have been found for the File_Type 'GIP_L2WATV' and for the satellite 'SENTINEL-2B'.
2020-04-02T09:02:40.833214 janwevers.uservm.rscloud.vito.be maja-processing-3.3.5 3.3 [000000018205] [E] ****************************************************************************************************** [vnsMajaMainProcessor.cxx:main:129]
2020-04-02T09:02:40.836307 janwevers.uservm.rscloud.vito.be maja-processing-3.3.5 3.3 [000000018205] [E] PreProcessing: vns::Data::ERROR: DataApplicationHandler(0x2006a00): DataApplicationHandler::GetGippFilename: More than one input filename have been found for the File_Type 'GIP_L2WATV' and for the satellite 'SENTINEL-2B'.
2020-04-02T09:02:40.839316 janwevers.uservm.rscloud.vito.be maja-processing-3.3.5 3.3 [000000018205] [E] [vnsMajaMainProcessor.cxx:main:129]
2020-04-02T09:02:40.842070 janwevers.uservm.rscloud.vito.be maja-processing-3.3.5 3.3 [000000018205] [E] ****************************************************************************************************** [vnsMajaMainProcessor.cxx:main:129]
2020-04-02 09:02:40,879 - Start-Maja - INFO - #######################################
2020-04-02 09:02:40,879 - Start-Maja - INFO - Error detected, see: /home/janwevers/worldcover/proc/out/maja/Hamburg/32UNE/../gipp/SENTINEL2_NATIF//S2B_MSIL1C_20190714T103029_N0208_R108_T32UNE_20190714T122554.SAFE.log
2020-04-02 09:02:40,879 - Start-Maja - INFO - #######################################
The file S2B_MSIL1C_20190714T103029_N0208_R108_T32UNE_20190714T122554.SAFE.log is attached.
S2B_MSIL1C_20190714T103029_N0208_R108_T32UNE_20190714T122554.SAFE.log
I am running Start-MAJA on some data in Indonesia. I have downloaded the SWDB data from EarthExplorer using the coordinates from getCooordinates.py as AoI. In total I got three files: e111s03, e112s03 and e112s02. When I run Start-MAJA complains because one of the files (e111s02) is missing and fails:
File "/home/neutral/Start-MAJA/prepare_dtm/DTMCreation.py", line 326, in <module>
creator.run(args.out, args.tempout)
File "/home/neutral/Start-MAJA/prepare_dtm/DTMCreation.py", line 286, in run
water_zipped = self.WaterZipped)
File "/home/neutral/Start-MAJA/prepare_dtm/../prepare_dtm/tuilage_mnt_eau_S2.py", line 203, in run
Found SRTM zip-archives...
Found Water zip-archives...
Processing with GranuleID
49MFT
UTM49S
32749
EPSG:32749
0
0
0
0
109800
109800
0
600000
9800020
Working directory: /tmp/49MFTszlQPC
('/home/neutral/DTM_SWBD/Creation/SRTM', '/tmp/49MFTszlQPC', '/home/neutral/DTM_SWBD/Creation/Water', '/tmp/49MFTszlQPC')
[59, 13]
[59, 13]
((111.89908281566626, -1.809051653338671, 0.0), (112.88731792789117, -2.8011262590637003, 0.0))
([59, 13], [59, 13])
['srtm_59_13.tif']
((111.89908281566626, -1.809051653338671, 0.0), (112.88731792789117, -2.8011262590637003, 0.0))
([111, -2], [112, -3])
('longitudes', 111, 112)
('latitudes', -3, -2)
('center coordinates', [[111.5, -2.5], [111.5, -1.5], [112.5, -2.5], [112.5, -1.5]])
['e111s03', 'e111s02', 'e112s03', 'e112s02']
/tmp/49MFTszlQPC
('liste_fic_mnt', ['srtm_59_13.tif'])
FIC: srtm_59_13.tif
(<type 'str'>, <type 'str'>)
/home/neutral/DTM_SWBD/Creation/SRTM/srtm_59_13.tif
gdalwarp -r cubic -srcnodata -32767 -dstnodata 0 /tmp/49MFTszlQPC/tmpISoslZ/srtm_59_13.tif /tmp/49MFTszlQPC/tmpISoslZ/srtm_59_13nodata0.tif
e111s03
('#############Fichier eau :', '/tmp/49MFTszlQPC/tmpISoslZ/e111s03e.shp')
gdal_rasterize -burn 1 -l e111s03e /tmp/49MFTszlQPC/tmpISoslZ/e111s03e.shp /tmp/49MFTszlQPC/tmpISoslZ/srtm_59_13_tmp.tif
e111s02
('missing SWBD watr file : ', 'e111s02')
calcul_masque_eau_mnt, working_dir=working_dir)
File "/home/neutral/Start-MAJA/prepare_dtm/../prepare_dtm/lib_mnt.py", line 524, in fusion_mnt
land = TestLand(liste_centre_eau[i][0], liste_centre_eau[i][1])
File "/home/neutral/Start-MAJA/prepare_dtm/../prepare_dtm/lib_mnt.py", line 57, in TestLand
layer = dataSource.GetLayer()
AttributeError: 'NoneType' object has no attribute 'GetLayer'
1
Is there a way to continue processing even if there is not SWBD data?
Regards
Dear Olivier
It seems that we experienced a kind of "ramdom" error using Start-MAJA. We used to process a full year time series but for some reasons, the process is sometimes stopped with the following message:
[vnsMajaMainProcessor.cxx:main:130]
2019-11-05T00:33:44.090276 gagarine maja-processing-3.3.0 3.3 [000000022004] [E] ScientificProcessing: vns::Data::ERROR: : Impossible to detect a L2 product in the directory </home/kermap/Documents/Start-MAJA-master/MAJA_tmp/36RXV/36RXV/GIPP_S2AS2B_xxx///in>. [vnsL2ImageFileReaderFactory.txx:211] [MAJA Data Exception] [vnsMajaMainProcessor.cxx:main:130]
This issue makes me think that it could be a similar problem than this post : olivierhagolle/Start_maja#29
Have you any idea ?
Our server is running with ubuntu 18.04 and the temporary folder is on the local drive.
Thank you very much!
Best regards
Antoine
Below the last lines before the error
11-05 00:23:35,902 - Start-Maja - INFO - #################################
2019-11-05 00:23:35,902 - Start-Maja - INFO - #################################
2019-11-05 00:23:35,902 - Start-Maja - INFO - processing /home/kermap/SENTINEL-2/Antoine/L1C/tmp/36RXV/S2A_MSIL1C_20180526T081601_N0206_R121_T36RXV_20180526T120617.SAFE in nominal mode
2019-11-05 00:23:35,902 - Start-Maja - INFO - MAJA logfile: /home/kermap/SENTINEL-2/Antoine/L2A_2018/tmp/36RXV/36RXV/GIPP_S2AS2B_xxx///S2A_MSIL1C_20180526T081601_N0206_R121_T36RXV_20180526T120617.SAFE.log
2019-11-05 00:23:35,903 - Start-Maja - INFO - #################################
2019-11-05T00:33:44.035961 gagarine maja-processing-3.3.0 3.3 [000000022004] [E] vns::Data::ERROR: : Impossible to detect a L2 product in the directory </home/kermap/Documents/Start-MAJA-master/MAJA_tmp/36RXV/36RXV/GIPP_S2AS2B_xxx///in>. [vnsL2ImageFileReaderFactory.txx:GetListOfL2Products:211]
2019-11-05T00:33:44.084066 gagarine maja-processing-3.3.0 3.3 [000000022004] [E] ****************************************************************************************************** [vnsMajaMainProcessor.cxx:main:130]
2019-11-05T00:33:44.090276 gagarine maja-processing-3.3.0 3.3 [000000022004] [E] ScientificProcessing: vns::Data::ERROR: : Impossible to detect a L2 product in the directory </home/kermap/Documents/Start-MAJA-master/MAJA_tmp/36RXV/36RXV/GIPP_S2AS2B_xxx///in>. [vnsL2ImageFileReaderFactory.txx:211] [MAJA Data Exception] [vnsMajaMainProcessor.cxx:main:130]
2019-11-05T00:33:44.099892 gagarine maja-processing-3.3.0 3.3 [000000022004] [E] [vnsMajaMainProcessor.cxx:main:130]
2019-11-05T00:33:44.112144 gagarine maja-processing-3.3.0 3.3 [000000022004] [E] ****************************************************************************************************** [vnsMajaMainProcessor.cxx:main:130]
2019-11-05 00:33:44,547 - Start-Maja - INFO - #######################################
2019-11-05 00:33:44,547 - Start-Maja - INFO - Error detected, see: /home/kermap/SENTINEL-2/Antoine/L2A_2018/tmp/36RXV/36RXV/GIPP_S2AS2B_xxx///S2A_MSIL1C_20180526T081601_N0206_R121_T36RXV_20180526T120617.SAFE.log
2019-11-05 00:33:44,547 - Start-Maja - INFO - ##################################
Hello,
I updated sen2agri platform to 2.0.1, but maja doesn't process l2a product.The log file are attached
demmaccs.log
demmaccs_29SNR.log
When i run demmaccs manually, it gives Error while copying the filename <OUT_DIR/share/config/Templates/PMC_LxREPT.EEF> to </mnt/archive/maccs_def/morocco/l2a/S2B_MSIL2A_20191007T112119_N0208_R037_T29SNR_20191007T140832.SAFE/maccs_29SNR/PMC_LxREPT.EEF> ! [vnsSystem.cxx:CopyFile:406]
where outdir is /opt/MAJA/
Note that /opt/MAJA/ has a full right access. When i installed maja in /home/ and i run the command it gives:
File "/usr/share/sen2agri/sen2agri-demmaccs/demmaccs.py", line 552, in
out = maccs_launcher(demmaccs_contexts[0])
File "/usr/share/sen2agri/sen2agri-demmaccs/demmaccs.py", line 331, in maccs_launcher
if run_command(cmd_array, demmaccs_context.output, tile_log_filename) != 0:
File "/usr/lib/python2.7/site-packages/sen2agri_common_db.py", line 155, in run_command
res = subprocess.call(cmd_array, shell=False)
File "/usr/lib64/python2.7/subprocess.py", line 524, in call
return Popen(*popenargs, **kwargs).wait()
File "/usr/lib64/python2.7/subprocess.py", line 711, in init
errread, errwrite)
File "/usr/lib64/python2.7/subprocess.py", line 1327, in _execute_child
raise child_exception
OSError: [Errno 13] Permission denied
What might cause this error?Should maja be installed in opt or in another folder?
subj
Hi,
Just to tell you that there is a bad behaviour with GDAL 3 (at least <= 3.0.2).
In DTMCreation/tuilage_mnt_eau_S2.py, the coordinates tranformation from osr (line 104/105) returns opposite values : (y, x) instead of (x,y).
The consequence is a wrong intersection between S2 image and SRTM coordinates (wrong path of srtm file, line 118/119).
Issue from OSGEO already identified : OSGeo/gdal#1546
For now, MAJA works fine with GDAL 2, my advice is do not update to GDAL 3 until issue is fixed.
Nicolas Ekicier
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.