Comments (21)
olivierhagolle
commented on June 11, 2024
Hi,
Sorry for the issue, could you please try to remove everything in the repWork folder ( /mnt/passesData/MAJAConfFold/MAJAtemp/*), and try again ? This may happen when a MAJA processing is started and canceled (we should do something to protect that...)
Best regards,
Olivier
from maja.
RubenValDin
commented on June 11, 2024
Hello Olivier,
Thank you very much for your early reply. I have removed everything in the repWork folder, but I get the same error message.
$ python /mnt/passesData/RunMAJA/start_maja.py -v -f /mnt/passesData/RunMAJA/folders.txt -c GIPP_MAJA_3.1.2_MUSCATE -t 29TPH -s Spain -d 20180823 -e 20180902
2019-03-12 09:18:45,880 - Start-Maja - INFO - No existing L2 product, we start with backward mode
2019-03-12 09:18:45,880 - Start-Maja - INFO - => processing date 20180823
2019-03-12 09:18:45,880 - Start-Maja - INFO - dates to process in backward mode :
2019-03-12 09:18:45,880 - Start-Maja - INFO - -- 20180823 : /mnt/passesData/MAJAConfFold/L1C_PDGS/Spain/S2B_MSIL1C_20180823T112109_N0206_R037_T29TPH_20180823T152254.SAFE
2019-03-12 09:18:45,881 - Start-Maja - INFO - -- 20180828 : /mnt/passesData/MAJAConfFold/L1C_PDGS/Spain/S2A_MSIL1C_20180828T112111_N0206_R037_T29TPH_20180828T135154.SAFE
2019-03-12 09:18:45,881 - Start-Maja - INFO - -- 20180902 : /mnt/passesData/MAJAConfFold/L1C_PDGS/Spain/S2B_MSIL1C_20180902T112109_N0206_R037_T29TPH_20180902T140408.SAFE
Traceback (most recent call last):
File "/mnt/passesData/RunMAJA/start_maja.py", line 613, in <module>
start_maja(folder_file, context, site, tile, orbit, nb_backward, options, debug_mode)
File "/mnt/passesData/RunMAJA/start_maja.py", line 449, in start_maja
add_parameter_files(repGipp, repWork + "/in/", tile, repCams)
File "/mnt/passesData/RunMAJA/start_maja.py", line 158, in add_parameter_files
os.symlink(fic, repWorkIn + '/' + base)
OSError: [Errno 17] File exists
Also, I forgot to comment you yesterday that there is a bug in start_maja.py. When I run it the first time I got the following error message.
$ python /mnt/passesData/RunMAJA/start_maja.py -v -f /mnt/passesData/RunMAJA/folders.txt -c GIPP_MAJA_3.1.2_MUSCATE -t 29TPH -s Spain -d 20180823 -e 20180902
Traceback (most recent call last):
File "/mnt/passesData/RunMAJA/start_maja.py", line 40, in <module>
from convert_CAMS_DBL import exocam_creation
ImportError: No module named convert_CAMS_DBL
In order to run it, I had to comment out line 40 from convert_CAMS_DBL import exocam_creation, but just to clarify I am not using CAMS now.
Thank you very much
Regards
from maja.
olivierhagolle
commented on June 11, 2024
I was in a conference these last days, I will try to reproduce this error tomorrow
Olivier
from maja.
olivierhagolle
commented on June 11, 2024
When I use start-maja, I always launch the command from the directory where Maja is installed. Maybe our tool is not robust to the use from another directory. If you have time to test before me, please do...
Olivier
from maja.
RubenValDin
commented on June 11, 2024
Olivier,
I have launched start_maja.py from the directory where MAJA is installed, but I continue getting the same error message.
/opt/maja/3.2.2/bin$ python /mnt/passesData/RunMAJA/start_maja.py -f /mnt/passesData/RunMAJA/folders.txt -c GIPP_MAJA_3.1.2_MUSCATE -t 29TPH -s Spain -d 20180823 -e 20180902
2019-03-15 09:00:49,842 - Start-Maja - INFO - No existing L2 product, we start with backward mode
2019-03-15 09:00:49,842 - Start-Maja - INFO - => processing date 20180823
2019-03-15 09:00:49,842 - Start-Maja - INFO - dates to process in backward mode :
2019-03-15 09:00:49,842 - Start-Maja - INFO - -- 20180823 : /mnt/passesData/MAJAConfFold/L1C_PDGS/Spain/S2B_MSIL1C_20180823T112109_N0206_R037_T29TPH_20180823T152254.SAFE
2019-03-15 09:00:49,842 - Start-Maja - INFO - -- 20180828 : /mnt/passesData/MAJAConfFold/L1C_PDGS/Spain/S2A_MSIL1C_20180828T112111_N0206_R037_T29TPH_20180828T135154.SAFE
2019-03-15 09:00:49,842 - Start-Maja - INFO - -- 20180902 : /mnt/passesData/MAJAConfFold/L1C_PDGS/Spain/S2B_MSIL1C_20180902T112109_N0206_R037_T29TPH_20180902T140408.SAFE
Traceback (most recent call last):
File "/mnt/passesData/RunMAJA/start_maja.py", line 613, in <module>
start_maja(folder_file, context, site, tile, orbit, nb_backward, options, debug_mode)
File "/mnt/passesData/RunMAJA/start_maja.py", line 449, in start_maja
add_parameter_files(repGipp, repWork + "/in/", tile, repCams)
File "/mnt/passesData/RunMAJA/start_maja.py", line 158, in add_parameter_files
os.symlink(fic, repWorkIn + '/' + base)
OSError: [Errno 17] File exists
Thanks
Ruben
from maja.
olivierhagolle
commented on June 11, 2024
Sorry, I still don't understand what's happening, and I do not observe that in my implementation.
By the way, did you notice that the whole Spain is processed by Theia, and is available for download from https://theia.cnes.fr
Just to be sure, are you using the start-maja version provided in this CNES repository, not the old one under my repository ? Because I thought I had corrected the issue with from convert_CAMS_DBL import exocam_creation
Could you please provide the content of folders.txt ?
Best regards,
Olivier
from maja.
RubenValDin
commented on June 11, 2024
Yes, I have noticed that the whole Spain is processed by Theia, but what I am doing is a test so I would like to be able to get Sentinel-2 data processed by myself using MAJA.
I am using the start_maja.py provided in the CNES repository. However, the issue with from convert_CAMS_DBL import exocam_creation is still in start_maja.py in this repository.
https://github.com/CNES/Start-MAJA/blob/1d4de564d865987734f2a2132fcab59be592216c/start_maja.py#L40
The content of the folders.txt was provided in the first comment.
I will continue trying it, perhaps I am doing something wrong and I haven't seen it yet.
Kind regards
from maja.
olivierhagolle
commented on June 11, 2024
Hi and thanks,
regarding the "file exists" bug, I have added a test for each symbolic link creation, if the file exists, it is not created again. It might work, but I am not sure, and I can't test as I am not able to reproduce the error.
One over point, it seems you installed MAJA as root (/opt...). This is not necessary, and caused problems to another user (see #3). He solved it by installing MAJA as a standard user.
Best regards,
Olivier
from maja.
olivierhagolle
commented on June 11, 2024
Moreover, I have updated start-MAJA to solve the import bug. If you want to update it.
Best regards
Olivier
from maja.
RubenValDin
commented on June 11, 2024
Olivier,
Thank you very much for updating Start-MAJA. I have added sys.exit(-3) into the python script following the next issue: #5 (comment), and I don't get the previous error. However, I am getting a different one:
~/InstallMAJA/bin$ python /mnt/passesData/RunMAJA/start_maja.py -f /mnt/passesData/RunMAJA/folders.txt -c GIPP_MAJA_3.1.2_MUSCATE -t 29TPH -s Spain -d 20180823 -e 20180902
2019-03-18T15:01:31.832900 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [I] MAJA Software Version : 3.2.2
2019-03-18T15:01:31.833861 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [I] Overloads the conf value in the command line by '/mnt/passesData/MAJAConfFold/MAJAtemp/Spain/29TPH/GIPP_MAJA_3.1.2_MUSCATE//userconf'.
2019-03-18T15:01:31.837575 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [I] -------------------------------------------------------------------------
2019-03-18T15:01:31.837734 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [I] Configuration User System parameters:
2019-03-18T15:01:31.838330 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [I] * L2Note : L2 note
2019-03-18T15:01:31.838768 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [I] * L3Note : L3 note
2019-03-18T15:01:31.839167 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [I] * CheckToolsNote : Checktool note
2019-03-18T15:01:31.839578 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [I] * NbThreads : 8
2019-03-18T15:01:31.840014 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [I] * EnableCleaningTemporaryDirectory : true
2019-03-18T15:01:31.840418 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [I] * EnableCachingConvertReflectanceData : true
2019-03-18T15:01:31.840814 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [I] * EnableCleaningCachingDirectoryBeforeProcessing : false
2019-03-18T15:01:31.841229 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [I] * EnableCleaningCachingDirectoryAfterProcessing : true
2019-03-18T15:01:31.841645 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [I] * EnableL2ResolutionProcessing : true
2019-03-18T15:01:31.842047 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [I] * PreProcessingCleanInputZipFiles : false
2019-03-18T15:01:31.842445 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [I] * PostProcessingZipFiles : false
2019-03-18T15:01:31.842840 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [I] * PostProcessingCleanFiles : false
2019-03-18T15:01:31.843258 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [I] * CheckXMLFilesWithSchema : false
2019-03-18T15:01:31.843651 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [I] -------------------------------------------------------------------------
2019-03-18T15:01:31.844172 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [I] Using default Admin configuration system file.
2019-03-18T15:01:31.847239 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [I] -------------------------------------------------------------------------
2019-03-18T15:01:31.847376 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [I] Configuration Admin System parameters:
2019-03-18T15:01:31.847817 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [I] * System : MAJA
2019-03-18T15:01:31.848324 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [I] * Tolerances / Footprint / Coordinate : 1e-06
2019-03-18T15:01:31.848731 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [I] * '' '' / Direction : 1e-06
2019-03-18T15:01:31.849177 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [I] * EnablePerformanceMeasureAlgorithms : false
2019-03-18T15:01:31.849673 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [I] -------------------------------------------------------------------------
2019-03-18T15:01:31.850288 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [I] ******************************************************************************
2019-03-18T15:01:31.850704 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [I] Application information:
2019-03-18T15:01:31.851136 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [I] * Processor Name : MAJA_L2_BACKWARD_CHAIN
2019-03-18T15:01:31.851573 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [I] * Test mode : false
2019-03-18T15:01:31.852069 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [I] * UserConfigSystemFilename : /mnt/passesData/MAJAConfFold/L2A_MAJA/Spain/29TPH/GIPP_MAJA_3.1.2_MUSCATE//.maja-working-directory/MAJAUserConfigSystem.xml
2019-03-18T15:01:31.852533 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [I] * AdminConfigSystemFileName : /mnt/passesData/MAJAConfFold/L2A_MAJA/Spain/29TPH/GIPP_MAJA_3.1.2_MUSCATE//.maja-working-directory/MAJAAdminConfigSystem.xml
2019-03-18T15:01:31.852985 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [I] * Specifics command line parameters:
2019-03-18T15:01:31.853429 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [I] - TileId: 29TPH
2019-03-18T15:01:31.854030 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [I] ******************************************************************************
2019-03-18T15:01:31.854586 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [I] The process uses 8 thread(s). Note: possible to use of maximum 128 threads.
2019-03-18T15:01:31.855161 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [I] Starting L2BackwardProcessor::PreProcessing() ...
2019-03-18T15:01:31.902987 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [I] There are 3 L1 images products.
2019-03-18T15:01:31.903186 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [I] - /mnt/passesData/MAJAConfFold/MAJAtemp/Spain/29TPH/GIPP_MAJA_3.1.2_MUSCATE//in/S2B_MSIL1C_20180823T112109_N0206_R037_T29TPH_20180823T152254.SAFE/MTD_MSIL1C.xml
2019-03-18T15:01:31.903735 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [I] - /mnt/passesData/MAJAConfFold/MAJAtemp/Spain/29TPH/GIPP_MAJA_3.1.2_MUSCATE//in/S2A_MSIL1C_20180828T112111_N0206_R037_T29TPH_20180828T135154.SAFE/MTD_MSIL1C.xml
2019-03-18T15:01:31.904201 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [I] - /mnt/passesData/MAJAConfFold/MAJAtemp/Spain/29TPH/GIPP_MAJA_3.1.2_MUSCATE//in/S2B_MSIL1C_20180902T112109_N0206_R037_T29TPH_20180902T140408.SAFE/MTD_MSIL1C.xml
2019-03-18T15:01:31.941347 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [P] -----------------------------------------------------------------------------------------------
2019-03-18T15:01:31.941552 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [P] Starting L2Processor::PreProcessing() ...
2019-03-18T15:01:31.942300 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [I] List of the 2 satellite(s) activated :
2019-03-18T15:01:31.942943 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [I] - SENTINEL-2A ('common' satellite SENTINEL-2_ and plugin SENTINEL2_TM)
2019-03-18T15:01:31.943867 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [I] - SENTINEL-2B ('common' satellite SENTINEL-2_ and plugin SENTINEL2_TM)
2019-03-18T15:01:31.944443 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [I] Starting reading and checking input data common for each plugin activated...
2019-03-18T15:01:31.944938 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [I] According to the input data, there is(are) 1 plugin(s) activated:
2019-03-18T15:01:31.945420 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [I] - SENTINEL2_TM
2019-03-18T15:01:31.945897 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [I] - SENTINEL2_TM
2019-03-18T15:01:31.947775 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [I] Starting DEM reading data for the 'unique' (common) satellite 'SENTINEL-2_' ...
2019-03-18T15:01:31.947951 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [I] UnCompress the DBL product </mnt/passesData/MAJAConfFold/MAJAtemp/Spain/29TPH/GIPP_MAJA_3.1.2_MUSCATE/in/S2__TEST_AUX_REFDE2_T29TPH_0001.DBL> ...
2019-03-18T15:01:31.948987 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [I] The DBL product </mnt/passesData/MAJAConfFold/MAJAtemp/Spain/29TPH/GIPP_MAJA_3.1.2_MUSCATE/in/S2__TEST_AUX_REFDE2_T29TPH_0001.DBL> is already uncompressed (.DBL.DIR directory already exists).
2019-03-18T15:01:31.950053 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [I] l_NumberOfL2Resolution : 2
2019-03-18T15:01:31.950358 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [I] Starting DTM reading ...
2019-03-18T15:01:31.962476 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [I] UnCompress the DBL product </mnt/passesData/MAJAConfFold/MAJAtemp/Spain/29TPH/GIPP_MAJA_3.1.2_MUSCATE/in/S2__TEST_AUX_REFDE2_T29TPH_0001.DBL> ...
2019-03-18T15:01:31.962740 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [I] The DBL product </mnt/passesData/MAJAConfFold/MAJAtemp/Spain/29TPH/GIPP_MAJA_3.1.2_MUSCATE/in/S2__TEST_AUX_REFDE2_T29TPH_0001.DBL> is already uncompressed (.DBL.DIR directory already exists).
2019-03-18T15:01:31.963546 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [I] Starting DTM reading (nb resolutions=2) ...
2019-03-18T15:01:37.912102 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [I] In the input data directory, there is(are) 0 meteo filename (EXO_METDTA).
2019-03-18T15:01:37.913003 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [I] In the input data directory, there is(are) 0 cams filename (EXO_CAMS).
2019-03-18T15:01:37.913221 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [I] Starting reading and checking input data for all satellites activated...
2019-03-18T15:01:37.913988 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [I] Initialize the Plugin instance detected with the Plugin Name 'SENTINEL2_TM' ...
2019-03-18T15:01:37.917989 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [I] The number of streaming lines of buffer for reading/writing images is '800'.
2019-03-18T15:01:37.918246 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [I] Using default Admin configuration camera file.
2019-03-18T15:01:37.922229 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [I] Starting reading data for the satellite 'SENTINEL-2A' and the plugin used is 'SENTINEL2_TM' ...
2019-03-18T15:01:37.922466 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [P] Starting GIP_L2SMAC reading ...
2019-03-18T15:01:37.924187 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [E] vns::Data::ERROR: DataApplicationHandler(0x1ab77f0): No GIPP of type GIP_L2SMAC has been detected for the Mission <SENTINEL-2A> in the input directory! [vnsDataApplicationHandler.cxx:GetListOfGippFilenames:1501]
2019-03-18T15:01:37.924575 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [E] ****************************************************************************************************** [vnsMajaMainProcessor.cxx:main:129]
2019-03-18T15:01:37.925026 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [E] PreProcessing: vns::Data::ERROR: DataApplicationHandler(0x1ab77f0): No GIPP of type GIP_L2SMAC has been detected for the Mission <SENTINEL-2A> in the input directory! [vnsDataApplicationHandler.cxx:1501] [MAJA Data Exception] [vnsMajaMainProcessor.cxx:main:129]
2019-03-18T15:01:37.925562 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [E] [vnsMajaMainProcessor.cxx:main:129]
2019-03-18T15:01:37.926153 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008364] [E] ****************************************************************************************************** [vnsMajaMainProcessor.cxx:main:129]
It seems that cannot detect GIP_L2SMAC in the input directory, but when I check the input directory the file is there.
in/
in/S2B_MSIL1C_20180823T112109_N0206_R037_T29TPH_20180823T152254.SAFE
in/S2A_TEST_GIP_CKEXTL_S_31TJF____10001_20150703_21000101.EEF
in/S2B_TEST_GIP_L2TOCR_L_ORGANICM_50001_00000000_99999999.HDR
in/S2A_TEST_GIP_L2COMM_L_ALLSITES_10009_20150703_21000101.EEF
in/S2B_TEST_GIP_CKEXTL_S_31TJF____10001_20150703_21000101.EEF
in/S2B_TEST_GIP_L2ALBD_L_ORGANICM_50001_00000000_99999999.HDR
in/S2B_TEST_GIP_L2ALBD_L_CONTINEN_10003_20150703_21000101.HDR
in/S2A_TEST_GIP_L2DIRT_L_BLACKCAR_50001_00000000_99999999.HDR
in/S2A_TEST_GIP_L2TOCR_L_CONTINEN_10005_20150703_21000101.HDR
in/S2B_TEST_GIP_L2DIRT_L_BLACKCAR_50001_00000000_99999999.HDR
in/S2A_TEST_GIP_L2DIRT_L_SULPHATE_50001_00000000_99999999.HDR
in/S2A_TEST_GIP_L2WATV_L_CONTINEN_10006_20150703_21000101.HDR
in/S2__TEST_AUX_REFDE2_T29TPH_0001_ALC.TIF
in/S2__TEST_AUX_REFDE2_T29TPH_0001.HDR
in/S2A_TEST_GIP_L2ALBD_L_BLACKCAR_50001_00000000_99999999.HDR
in/S2A_TEST_GIP_L2TOCR_L_ORGANICM_50001_00000000_99999999.HDR
in/S2__TEST_AUX_REFDE2_T29TPH_0001_SLP_R2.TIF
in/S2A_TEST_GIP_L2DIRT_L_SEASALT__50001_00000000_99999999.HDR
in/S2B_TEST_GIP_L2DIRT_L_ORGANICM_50001_00000000_99999999.HDR
in/S2B_TEST_GIP_L2DIFT_L_BLACKCAR_50001_00000000_99999999.HDR
in/S2B_TEST_GIP_L2DIFT_L_ORGANICM_50001_00000000_99999999.HDR
in/S2__TEST_GIP_L2SITE_S_31TJF____10001_00000000_99999999.EEF
in/S2__TEST_AUX_REFDE2_T29TPH_0001_MSK.TIF
in/S2B_TEST_GIP_L2DIRT_L_CONTINEN_10002_20150703_21000101.HDR
in/S2__TEST_AUX_REFDE2_T29TPH_0001_ASP_R1.TIF
in/S2A_MSIL1C_20180828T112111_N0206_R037_T29TPH_20180828T135154.SAFE
in/S2B_TEST_GIP_L2DIRT_L_DUST_____50001_00000000_99999999.HDR
in/S2__TEST_AUX_REFDE2_T29TPH_0001_ALT_R2.TIF
in/S2A_TEST_GIP_L2ALBD_L_DUST_____50001_00000000_99999999.HDR
in/S2B_MSIL1C_20180902T112109_N0206_R037_T29TPH_20180902T140408.SAFE
in/S2A_TEST_GIP_L2DIFT_L_BLACKCAR_50001_00000000_99999999.HDR
in/S2A_TEST_GIP_L2DIFT_L_CONTINEN_10005_20150703_21000101.HDR
in/S2B_TEST_GIP_L2TOCR_L_SEASALT__50001_00000000_99999999.HDR
in/S2A_TEST_GIP_L2DIFT_L_SULPHATE_50001_00000000_99999999.HDR
in/S2__TEST_AUX_REFDE2_T29TPH_0001.DBL.DIR
in/S2B_TEST_GIP_L2ALBD_L_SEASALT__50001_00000000_99999999.HDR
in/S2B_TEST_GIP_L2SMAC_L_ALLSITES_10005_20150703_21000101.EEF
in/S2__TEST_AUX_REFDE2_T29TPH_0001_ALT_R1.TIF
in/readme.md
in/S2A_TEST_GIP_L2ALBD_L_SULPHATE_50001_00000000_99999999.HDR
in/S2B_TEST_GIP_L2COMM_L_ALLSITES_10009_20150703_21000101.EEF
in/S2B_TEST_GIP_L2DIRT_L_SULPHATE_50001_00000000_99999999.HDR
in/S2B_TEST_GIP_L2TOCR_L_DUST_____50001_00000000_99999999.HDR
in/S2B_TEST_GIP_L2DIFT_L_SULPHATE_50001_00000000_99999999.HDR
in/S2B_TEST_GIP_CKQLTL_S_31TJF____10005_20150703_21000101.EEF
in/S2A_TEST_GIP_L2SMAC_L_ALLSITES_10005_20150703_21000101.EEF
in/S2A_TEST_GIP_CKQLTL_S_31TJF____10005_20150703_21000101.EEF
in/S2A_TEST_GIP_L2TOCR_L_BLACKCAR_50001_00000000_99999999.HDR
in/S2A_TEST_GIP_L2TOCR_L_SEASALT__50001_00000000_99999999.HDR
in/S2A_TEST_GIP_L2TOCR_L_DUST_____50001_00000000_99999999.HDR
in/S2A_TEST_GIP_L2DIRT_L_ORGANICM_50001_00000000_99999999.HDR
in/S2B_TEST_GIP_L2ALBD_L_DUST_____50001_00000000_99999999.HDR
in/S2A_TEST_GIP_L2ALBD_L_SEASALT__50001_00000000_99999999.HDR
in/S2B_TEST_GIP_L2DIFT_L_CONTINEN_10002_20150703_21000101.HDR
in/S2B_TEST_GIP_L2WATV_L_CONTINEN_10006_20150703_21000101.HDR
in/S2__TEST_AUX_REFDE2_T29TPH_0001_ASC.TIF
in/S2B_TEST_GIP_L2TOCR_L_SULPHATE_50001_00000000_99999999.HDR
in/S2B_TEST_GIP_L2TOCR_L_CONTINEN_10002_20150703_21000101.HDR
in/S2B_TEST_GIP_L2DIRT_L_SEASALT__50001_00000000_99999999.HDR
in/S2B_TEST_GIP_L2DIFT_L_SEASALT__50001_00000000_99999999.HDR
in/S2B_TEST_GIP_L2DIFT_L_DUST_____50001_00000000_99999999.HDR
in/S2A_TEST_GIP_L2DIRT_L_DUST_____50001_00000000_99999999.HDR
in/S2A_TEST_GIP_L2DIFT_L_SEASALT__50001_00000000_99999999.HDR
in/S2A_TEST_GIP_L2DIRT_L_CONTINEN_10005_20150703_21000101.HDR
in/S2B_TEST_GIP_L2TOCR_L_BLACKCAR_50001_00000000_99999999.HDR
in/S2A_TEST_GIP_L2ALBD_L_CONTINEN_10005_20150703_21000101.HDR
in/S2__TEST_AUX_REFDE2_T29TPH_0001_SLP_R1.TIF
in/S2B_TEST_GIP_L2ALBD_L_BLACKCAR_50001_00000000_99999999.HDR
in/S2A_TEST_GIP_L2DIFT_L_ORGANICM_50001_00000000_99999999.HDR
in/S2A_TEST_GIP_L2DIFT_L_DUST_____50001_00000000_99999999.HDR
in/S2A_TEST_GIP_L2ALBD_L_ORGANICM_50001_00000000_99999999.HDR
in/S2__TEST_AUX_REFDE2_T29TPH_0001_ASP_R2.TIF
in/S2A_TEST_GIP_L2TOCR_L_SULPHATE_50001_00000000_99999999.HDR
in/S2B_TEST_GIP_L2ALBD_L_SULPHATE_50001_00000000_99999999.HDR
in/S2__TEST_AUX_REFDE2_T29TPH_0001_SLC.TIF
Kind regards
from maja.
olivierhagolle
commented on June 11, 2024
oops I should have seen that way before, but only saw it now, sorry
It looks like you have installed the MAJA-TM version, but you are using the parameters that go along the No-TM version, which causes the errors with parameters.
You are right to use the TM version, but the parameters must be taken from
http://tully.ups-tlse.fr/olivier/gipp_maja/tree/master/GIPP_MAJA_3.1.2_TM
and not from http://tully.ups-tlse.fr/olivier/gipp_maja/tree/master/GIPP_MAJA_3.1.2_MUSCATE
You will have also to get the data from the zenodo server indicated in the readme.md of GIPP_MAJA_3.1.2_TM
You might wonder why is that, but, really, that's very weird (let's say it is for historuc reasons ;)). Anyway, the field in each GIPP text file either contains SENTINEL-2A (MUSCATE plugin) or SENTINEL2A with a dash, with TM plugin.
So your error message means that it searches SENTINEL2A in the GGIPP file and does not find it.
from maja.
olivierhagolle
commented on June 11, 2024
And the line you added in start_maja is not necessary, please remove it, it was just to stop execution to print what happens. But you have a different issue.
from maja.
RubenValDin
commented on June 11, 2024
Hello Olivier,
I didn't realize about the GIPP_MAJA_3.1.2_TM and GIPP_MAJA_3.1.2_MUSCATE. I have downloaded the TM from the link provided, however, not sure about the data from the zenodo server, those are the LUT but if I am not using CAMS then I wouldn't need these files.
I have deleted the line I had added and run start_maja.py without that line and the GIPP_MAJA_3.1.2_TM, but I get the following error message:
2019-03-19T09:22:38.444618 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000007435] [I] MAJA Software Version : 3.2.2
2019-03-19T09:22:38.445460 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000007435] [I] Overloads the conf value in the command line by '/mnt/passesData/MAJAConfFold/MAJAtemp/Spain/29TPH/GIPP_MAJA_3.1.2_TM//userconf'.
2019-03-19T09:22:38.557836 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000007435] [I] -------------------------------------------------------------------------
2019-03-19T09:22:38.559982 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000007435] [I] Configuration User System parameters:
2019-03-19T09:22:38.563405 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000007435] [I] * L2Note : L2 note
2019-03-19T09:22:38.567151 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000007435] [I] * L3Note : L3 note
2019-03-19T09:22:38.571153 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000007435] [I] * CheckToolsNote : Checktool note
2019-03-19T09:22:38.574457 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000007435] [I] * NbThreads : 8
2019-03-19T09:22:38.578823 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000007435] [I] * EnableCleaningTemporaryDirectory : true
2019-03-19T09:22:38.582693 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000007435] [I] * EnableCachingConvertReflectanceData : true
2019-03-19T09:22:38.585462 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000007435] [I] * EnableCleaningCachingDirectoryBeforeProcessing : false
2019-03-19T09:22:38.589211 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000007435] [I] * EnableCleaningCachingDirectoryAfterProcessing : true
2019-03-19T09:22:38.592852 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000007435] [I] * EnableL2ResolutionProcessing : true
2019-03-19T09:22:38.596058 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000007435] [I] * PreProcessingCleanInputZipFiles : false
2019-03-19T09:22:38.599652 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000007435] [I] * PostProcessingZipFiles : false
2019-03-19T09:22:38.600147 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000007435] [I] * PostProcessingCleanFiles : false
2019-03-19T09:22:38.602786 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000007435] [I] * CheckXMLFilesWithSchema : false
2019-03-19T09:22:38.606121 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000007435] [I] -------------------------------------------------------------------------
2019-03-19T09:22:38.609808 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000007435] [I] Using default Admin configuration system file.
2019-03-19T09:22:38.615716 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000007435] [I] -------------------------------------------------------------------------
2019-03-19T09:22:38.616240 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000007435] [I] Configuration Admin System parameters:
2019-03-19T09:22:38.620070 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000007435] [I] * System : MAJA
2019-03-19T09:22:38.624088 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000007435] [I] * Tolerances / Footprint / Coordinate : 1e-06
2019-03-19T09:22:38.627547 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000007435] [I] * '' '' / Direction : 1e-06
2019-03-19T09:22:38.631244 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000007435] [I] * EnablePerformanceMeasureAlgorithms : false
2019-03-19T09:22:38.634966 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000007435] [I] -------------------------------------------------------------------------
2019-03-19T09:22:38.638380 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000007435] [I] ******************************************************************************
2019-03-19T09:22:38.642043 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000007435] [I] Application information:
2019-03-19T09:22:38.646157 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000007435] [I] * Processor Name : MAJA_L2_BACKWARD_CHAIN
2019-03-19T09:22:38.649403 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000007435] [I] * Test mode : false
2019-03-19T09:22:38.652917 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000007435] [I] * UserConfigSystemFilename : /mnt/passesData/MAJAConfFold/L2A_MAJA/Spain/29TPH/GIPP_MAJA_3.1.2_TM//.maja-working-directory/MAJAUserConfigSystem.xml
2019-03-19T09:22:38.656614 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000007435] [I] * AdminConfigSystemFileName : /mnt/passesData/MAJAConfFold/L2A_MAJA/Spain/29TPH/GIPP_MAJA_3.1.2_TM//.maja-working-directory/MAJAAdminConfigSystem.xml
2019-03-19T09:22:38.803148 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000007435] [I] * Specifics command line parameters:
2019-03-19T09:22:38.806749 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000007435] [I] - TileId: 29TPH
2019-03-19T09:22:38.810748 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000007435] [I] ******************************************************************************
2019-03-19T09:22:38.814386 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000007435] [I] The process uses 8 thread(s). Note: possible to use of maximum 128 threads.
2019-03-19T09:22:38.817298 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000007435] [I] Starting L2BackwardProcessor::PreProcessing() ...
2019-03-19T09:22:38.821091 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000007435] [E] vns::Data::ERROR: : Impossible to detect a L1 product in the directory </mnt/passesData/MAJAConfFold/MAJAtemp/Spain/29TPH/GIPP_MAJA_3.1.2_TM//in>. [vnsL1ImageInformationsProviderFactory.cxx:GetListOfL1Products:150]
2019-03-19T09:22:38.824844 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000007435] [E] ****************************************************************************************************** [vnsMajaMainProcessor.cxx:main:129]
2019-03-19T09:22:38.827712 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000007435] [E] PreProcessing: vns::Data::ERROR: : Impossible to detect a L1 product in the directory </mnt/passesData/MAJAConfFold/MAJAtemp/Spain/29TPH/GIPP_MAJA_3.1.2_TM//in>. [vnsL1ImageInformationsProviderFactory.cxx:150] [MAJA Data Exception] [vnsMajaMainProcessor.cxx:main:129]
2019-03-19T09:22:38.830988 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000007435] [E] [vnsMajaMainProcessor.cxx:main:129]
2019-03-19T09:22:38.834469 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000007435] [E] ****************************************************************************************************** [vnsMajaMainProcessor.cxx:main:129]
from maja.
RubenValDin
commented on June 11, 2024
Olivier, I have done a test keeping sys.exit(-3) into the python script. I don't get the previous error, but I get a different one:
2019-03-19T09:39:01.298528 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [I] MAJA Software Version : 3.2.2
2019-03-19T09:39:01.299180 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [I] Overloads the conf value in the command line by '/mnt/passesData/MAJAConfFold/MAJAtemp/Spain/29TPH/GIPP_MAJA_3.1.2_TM//userconf'.
2019-03-19T09:39:01.302411 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [I] -------------------------------------------------------------------------
2019-03-19T09:39:01.302536 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [I] Configuration User System parameters:
2019-03-19T09:39:01.302998 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [I] * L2Note : L2 note
2019-03-19T09:39:01.303391 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [I] * L3Note : L3 note
2019-03-19T09:39:01.303779 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [I] * CheckToolsNote : Checktool note
2019-03-19T09:39:01.304174 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [I] * NbThreads : 8
2019-03-19T09:39:01.304551 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [I] * EnableCleaningTemporaryDirectory : true
2019-03-19T09:39:01.304920 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [I] * EnableCachingConvertReflectanceData : true
2019-03-19T09:39:01.305288 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [I] * EnableCleaningCachingDirectoryBeforeProcessing : false
2019-03-19T09:39:01.305693 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [I] * EnableCleaningCachingDirectoryAfterProcessing : true
2019-03-19T09:39:01.306075 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [I] * EnableL2ResolutionProcessing : true
2019-03-19T09:39:01.306459 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [I] * PreProcessingCleanInputZipFiles : false
2019-03-19T09:39:01.306847 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [I] * PostProcessingZipFiles : false
2019-03-19T09:39:01.307258 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [I] * PostProcessingCleanFiles : false
2019-03-19T09:39:01.307698 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [I] * CheckXMLFilesWithSchema : false
2019-03-19T09:39:01.308091 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [I] -------------------------------------------------------------------------
2019-03-19T09:39:01.308483 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [I] Using default Admin configuration system file.
2019-03-19T09:39:01.311562 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [I] -------------------------------------------------------------------------
2019-03-19T09:39:01.311700 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [I] Configuration Admin System parameters:
2019-03-19T09:39:01.312171 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [I] * System : MAJA
2019-03-19T09:39:01.312615 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [I] * Tolerances / Footprint / Coordinate : 1e-06
2019-03-19T09:39:01.313022 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [I] * '' '' / Direction : 1e-06
2019-03-19T09:39:01.313446 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [I] * EnablePerformanceMeasureAlgorithms : false
2019-03-19T09:39:01.313879 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [I] -------------------------------------------------------------------------
2019-03-19T09:39:01.314341 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [I] ******************************************************************************
2019-03-19T09:39:01.314776 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [I] Application information:
2019-03-19T09:39:01.315216 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [I] * Processor Name : MAJA_L2_BACKWARD_CHAIN
2019-03-19T09:39:01.315670 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [I] * Test mode : false
2019-03-19T09:39:01.316177 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [I] * UserConfigSystemFilename : /mnt/passesData/MAJAConfFold/L2A_MAJA/Spain/29TPH/GIPP_MAJA_3.1.2_TM//.maja-working-directory/MAJAUserConfigSystem.xml
2019-03-19T09:39:01.316786 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [I] * AdminConfigSystemFileName : /mnt/passesData/MAJAConfFold/L2A_MAJA/Spain/29TPH/GIPP_MAJA_3.1.2_TM//.maja-working-directory/MAJAAdminConfigSystem.xml
2019-03-19T09:39:01.317342 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [I] * Specifics command line parameters:
2019-03-19T09:39:01.317955 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [I] - TileId: 29TPH
2019-03-19T09:39:01.318528 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [I] ******************************************************************************
2019-03-19T09:39:01.319153 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [I] The process uses 8 thread(s). Note: possible to use of maximum 128 threads.
2019-03-19T09:39:01.319767 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [I] Starting L2BackwardProcessor::PreProcessing() ...
2019-03-19T09:39:01.423660 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [I] There are 3 L1 images products.
2019-03-19T09:39:01.423921 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [I] - /mnt/passesData/MAJAConfFold/MAJAtemp/Spain/29TPH/GIPP_MAJA_3.1.2_TM//in/S2B_MSIL1C_20180823T112109_N0206_R037_T29TPH_20180823T152254.SAFE/MTD_MSIL1C.xml
2019-03-19T09:39:01.424572 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [I] - /mnt/passesData/MAJAConfFold/MAJAtemp/Spain/29TPH/GIPP_MAJA_3.1.2_TM//in/S2A_MSIL1C_20180828T112111_N0206_R037_T29TPH_20180828T135154.SAFE/MTD_MSIL1C.xml
2019-03-19T09:39:01.425126 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [I] - /mnt/passesData/MAJAConfFold/MAJAtemp/Spain/29TPH/GIPP_MAJA_3.1.2_TM//in/S2B_MSIL1C_20180902T112109_N0206_R037_T29TPH_20180902T140408.SAFE/MTD_MSIL1C.xml
2019-03-19T09:39:01.463970 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [P] -----------------------------------------------------------------------------------------------
2019-03-19T09:39:01.464153 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [P] Starting L2Processor::PreProcessing() ...
2019-03-19T09:39:01.464676 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [I] List of the 2 satellite(s) activated :
2019-03-19T09:39:01.465187 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [I] - SENTINEL-2A ('common' satellite SENTINEL-2_ and plugin SENTINEL2_TM)
2019-03-19T09:39:01.465692 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [I] - SENTINEL-2B ('common' satellite SENTINEL-2_ and plugin SENTINEL2_TM)
2019-03-19T09:39:01.466166 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [I] Starting reading and checking input data common for each plugin activated...
2019-03-19T09:39:01.466643 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [I] According to the input data, there is(are) 1 plugin(s) activated:
2019-03-19T09:39:01.467162 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [I] - SENTINEL2_TM
2019-03-19T09:39:01.467661 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [I] - SENTINEL2_TM
2019-03-19T09:39:01.472757 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [I] Starting DEM reading data for the 'unique' (common) satellite 'SENTINEL-2_' ...
2019-03-19T09:39:01.472915 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [I] UnCompress the DBL product </mnt/passesData/MAJAConfFold/MAJAtemp/Spain/29TPH/GIPP_MAJA_3.1.2_TM/in/S2__TEST_AUX_REFDE2_T29TPH_0001.DBL> ...
2019-03-19T09:39:01.473488 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [I] The DBL product </mnt/passesData/MAJAConfFold/MAJAtemp/Spain/29TPH/GIPP_MAJA_3.1.2_TM/in/S2__TEST_AUX_REFDE2_T29TPH_0001.DBL> is already uncompressed (.DBL.DIR directory already exists).
2019-03-19T09:39:01.474266 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [I] l_NumberOfL2Resolution : 2
2019-03-19T09:39:01.474497 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [I] Starting DTM reading ...
2019-03-19T09:39:01.492183 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [I] UnCompress the DBL product </mnt/passesData/MAJAConfFold/MAJAtemp/Spain/29TPH/GIPP_MAJA_3.1.2_TM/in/S2__TEST_AUX_REFDE2_T29TPH_0001.DBL> ...
2019-03-19T09:39:01.492466 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [I] The DBL product </mnt/passesData/MAJAConfFold/MAJAtemp/Spain/29TPH/GIPP_MAJA_3.1.2_TM/in/S2__TEST_AUX_REFDE2_T29TPH_0001.DBL> is already uncompressed (.DBL.DIR directory already exists).
2019-03-19T09:39:01.493277 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [I] Starting DTM reading (nb resolutions=2) ...
2019-03-19T09:39:06.873737 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [I] In the input data directory, there is(are) 0 meteo filename (EXO_METDTA).
2019-03-19T09:39:06.874598 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [I] In the input data directory, there is(are) 0 cams filename (EXO_CAMS).
2019-03-19T09:39:06.874805 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [I] Starting reading and checking input data for all satellites activated...
2019-03-19T09:39:06.875534 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [I] Initialize the Plugin instance detected with the Plugin Name 'SENTINEL2_TM' ...
2019-03-19T09:39:06.888029 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [I] The number of streaming lines of buffer for reading/writing images is '800'.
2019-03-19T09:39:06.888275 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [I] Using default Admin configuration camera file.
2019-03-19T09:39:06.894383 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [I] Starting reading data for the satellite 'SENTINEL-2A' and the plugin used is 'SENTINEL2_TM' ...
2019-03-19T09:39:06.894670 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [P] Starting GIP_L2SMAC reading ...
2019-03-19T09:39:06.897002 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [P] Starting GIP_L2TOCR reading ...
2019-03-19T09:39:06.899164 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [I] UnCompress the DBL product </mnt/passesData/MAJAConfFold/MAJAtemp/Spain/29TPH/GIPP_MAJA_3.1.2_TM/in/S2A_TEST_GIP_L2TOCR_L_CONTINEN_10005_20150703_21000101.DBL> ...
2019-03-19T09:39:06.902665 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [E] vns::Data::ERROR: : Error while uncompressing the DBL product </mnt/passesData/MAJAConfFold/MAJAtemp/Spain/29TPH/GIPP_MAJA_3.1.2_TM/in/S2A_TEST_GIP_L2TOCR_L_CONTINEN_10005_20150703_21000101.DBL> in the destination directory </mnt/passesData/MAJAConfFold/MAJAtemp/Spain/29TPH/GIPP_MAJA_3.1.2_TM/in>!, Log of execution :Command line : tar -xf /mnt/passesData/MAJAConfFold/MAJAtemp/Spain/29TPH/GIPP_MAJA_3.1.2_TM/in/S2A_TEST_GIP_L2TOCR_L_CONTINEN_10005_20150703_21000101.DBL -C /mnt/passesData/MAJAConfFold/MAJAtemp/Spain/29TPH/GIPP_MAJA_3.1.2_TM/in .Log : tar: /mnt/passesData/MAJAConfFold/MAJAtemp/Spain/29TPH/GIPP_MAJA_3.1.2_TM/in/S2A_TEST_GIP_L2TOCR_L_CONTINEN_10005_20150703_21000101.DBL: Cannot open: No such file or directory
tar: Error is not recoverable: exiting now
[vnsSystem.cxx:UnCompressDBLProduct:255]
2019-03-19T09:39:06.905401 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [E] ****************************************************************************************************** [vnsMajaMainProcessor.cxx:main:129]
2019-03-19T09:39:06.905636 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [E] PreProcessing: vns::Data::ERROR: : Error while uncompressing the DBL product </mnt/passesData/MAJAConfFold/MAJAtemp/Spain/29TPH/GIPP_MAJA_3.1.2_TM/in/S2A_TEST_GIP_L2TOCR_L_CONTINEN_10005_20150703_21000101.DBL> in the destination directory </mnt/passesData/MAJAConfFold/MAJAtemp/Spain/29TPH/GIPP_MAJA_3.1.2_TM/in>!, Log of execution :Command line : tar -xf /mnt/passesData/MAJAConfFold/MAJAtemp/Spain/29TPH/GIPP_MAJA_3.1.2_TM/in/S2A_TEST_GIP_L2TOCR_L_CONTINEN_10005_20150703_21000101.DBL -C /mnt/passesData/MAJAConfFold/MAJAtemp/Spain/29TPH/GIPP_MAJA_3.1.2_TM/in .Log : tar: /mnt/passesData/MAJAConfFold/MAJAtemp/Spain/29TPH/GIPP_MAJA_3.1.2_TM/in/S2A_TEST_GIP_L2TOCR_L_CONTINEN_10005_20150703_21000101.DBL: Cannot open: No such file or directory
tar: Error is not recoverable: exiting now
[vnsSystem.cxx:255] [MAJA Data Exception] [vnsMajaMainProcessor.cxx:main:129]
2019-03-19T09:39:06.906244 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [E] [vnsMajaMainProcessor.cxx:main:129]
2019-03-19T09:39:06.906865 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000008116] [E] ****************************************************************************************************** [vnsMajaMainProcessor.cxx:main:129]
from maja.
olivierhagolle
commented on June 11, 2024
Hi Ruben,
yes, you need the LUT files too (they contain CAMS, but also the constant continental model). This is the message you get.
Olivier
from maja.
RubenValDin
commented on June 11, 2024
Hi Olivier,
Thank you very much. It worked with the LUT files, but I got the same error as #5 (it doesn't quit).
2019-03-20T14:12:51.643415 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000022436] [I] Cleans the caching directory (if not already deleted) </mnt/passesData/MAJAConfFold/L2A_MAJA/Spain/29TPH/GIPP_MAJA_3.1.2_TM//.maja-working-directory/caching>...
2019-03-20T14:12:51.646376 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000022436] [P] Starting Unload data application handler done.
2019-03-20T14:12:51.647153 ip-172-31-42-160 maja-processing-3.2.2 3.2 [000000022436] [I] Process OK ;-) [code return: 0]
Outputs in repL2 directory:
S2B_MSIL1C_20180823T112109_N0206_R037_T29TPH_20180823T152254.SAFE.log
S2B_OPER_PMC_L2REPT_29TPH____20180823.EEF
S2B_TEST_QCK_L2VALD_T29TPH_20180823.DBL.DIR
S2B_TEST_QCK_L2VALD_T29TPH_20180823.HDR
SENTINEL2B_20180823-112334-374_L2A_T29TPH_C_V1-0
And inside SENTINEL2B_20180823-112334-374_L2A_T29TPH_C_V1-0:
DATA
MASKS
SENTINEL2B_20180823-112334-374_L2A_T29TPH_C_V1-0_ATB_R1.tif
SENTINEL2B_20180823-112334-374_L2A_T29TPH_C_V1-0_ATB_R2.tif
SENTINEL2B_20180823-112334-374_L2A_T29TPH_C_V1-0_FRE_B11.tif
SENTINEL2B_20180823-112334-374_L2A_T29TPH_C_V1-0_FRE_B12.tif
SENTINEL2B_20180823-112334-374_L2A_T29TPH_C_V1-0_FRE_B2.tif
SENTINEL2B_20180823-112334-374_L2A_T29TPH_C_V1-0_FRE_B3.tif
SENTINEL2B_20180823-112334-374_L2A_T29TPH_C_V1-0_FRE_B4.tif
SENTINEL2B_20180823-112334-374_L2A_T29TPH_C_V1-0_FRE_B5.tif
SENTINEL2B_20180823-112334-374_L2A_T29TPH_C_V1-0_FRE_B6.tif
SENTINEL2B_20180823-112334-374_L2A_T29TPH_C_V1-0_FRE_B7.tif
SENTINEL2B_20180823-112334-374_L2A_T29TPH_C_V1-0_FRE_B8.tif
SENTINEL2B_20180823-112334-374_L2A_T29TPH_C_V1-0_FRE_B8A.tif
SENTINEL2B_20180823-112334-374_L2A_T29TPH_C_V1-0_MTD_ALL.xml
SENTINEL2B_20180823-112334-374_L2A_T29TPH_C_V1-0_QKL_ALL.jpg
SENTINEL2B_20180823-112334-374_L2A_T29TPH_C_V1-0_SRE_B11.tif
SENTINEL2B_20180823-112334-374_L2A_T29TPH_C_V1-0_SRE_B12.tif
SENTINEL2B_20180823-112334-374_L2A_T29TPH_C_V1-0_SRE_B2.tif
SENTINEL2B_20180823-112334-374_L2A_T29TPH_C_V1-0_SRE_B3.tif
SENTINEL2B_20180823-112334-374_L2A_T29TPH_C_V1-0_SRE_B4.tif
SENTINEL2B_20180823-112334-374_L2A_T29TPH_C_V1-0_SRE_B5.tif
SENTINEL2B_20180823-112334-374_L2A_T29TPH_C_V1-0_SRE_B6.tif
SENTINEL2B_20180823-112334-374_L2A_T29TPH_C_V1-0_SRE_B7.tif
SENTINEL2B_20180823-112334-374_L2A_T29TPH_C_V1-0_SRE_B8.tif
SENTINEL2B_20180823-112334-374_L2A_T29TPH_C_V1-0_SRE_B8A.tif
However the SAFE.log file is empty.
Regards
from maja.
olivierhagolle
commented on June 11, 2024
Hi,
Good, we are making progress :)
Sorry, I do not understand "It doesn't quit" (I can be dumb in the afternoon ;) )
The 3 lines of log displayed seem fine.
Lets'go back to basics :
- which command line did you use ?
- what was written by start-maja on the console ?
- what is in each .SAFE.log
Olivier
from maja.
RubenValDin
commented on June 11, 2024
Olivier,
The command line I have used is the following:
~/InstallMAJA/bin$ python /mnt/passesData/RunMAJA/start_maja.py -f /mnt/passesData/RunMAJA/folders.txt -c GIPP_MAJA_3.1.2_TM -t 29TPH -s Spain -d 20180823 -e 20180902
I have attached what was written by start-maja on the console.
What do you mean by each .SAFE.log? I have just got one which is empty.
Regards,
from maja.
olivierhagolle
commented on June 11, 2024
I just learned that some types of shells do not accept redirections as I did. I have updated start_maja.py using the portable version of stdout and stderr streams redirection. I am not sure it will solve all issues, but at least, Maja log will get into the logfile, and not on the console.
Could you please try it ?
from maja.
RubenValDin
commented on June 11, 2024
Olivier, sorry for the delay.
I have tested the update of start_maja.py and it works properly.
On the console I just get the next:
2019-03-25 09:18:50,242 - Start-Maja - INFO - No existing L2 product, we start with backward mode
2019-03-25 09:18:50,243 - Start-Maja - INFO - => processing date 20180823
2019-03-25 09:18:50,244 - Start-Maja - INFO - dates to process in backward mode :
2019-03-25 09:18:50,244 - Start-Maja - INFO - -- 20180823 : /mnt/passesData/MAJAConfFold/L1C_PDGS/Spain/S2B_MSIL1C_20180823T112109_N0206_R037_T29TPH_20180823T152254.SAFE
2019-03-25 09:18:50,244 - Start-Maja - INFO - -- 20180828 : /mnt/passesData/MAJAConfFold/L1C_PDGS/Spain/S2A_MSIL1C_20180828T112111_N0206_R037_T29TPH_20180828T135154.SAFE
2019-03-25 09:18:50,244 - Start-Maja - INFO - -- 20180902 : /mnt/passesData/MAJAConfFold/L1C_PDGS/Spain/S2B_MSIL1C_20180902T112109_N0206_R037_T29TPH_20180902T140408.SAFE
2019-03-25 09:18:50,255 - Start-Maja - INFO - #################################
2019-03-25 09:18:50,255 - Start-Maja - INFO - #################################
2019-03-25 09:18:50,255 - Start-Maja - INFO - processing /mnt/passesData/MAJAConfFold/L1C_PDGS/Spain/S2B_MSIL1C_20180823T112109_N0206_R037_T29TPH_20180823T152254.SAFE in backward mode
2019-03-25 09:18:50,255 - Start-Maja - INFO - Initialisation mode with backward is longer
2019-03-25 09:18:50,255 - Start-Maja - INFO - MAJA logfile: /mnt/passesData/MAJAConfFold/L2A_MAJA/Spain/29TPH/GIPP_MAJA_3.1.2_TM//S2B_MSIL1C_20180823T112109_N0206_R037_T29TPH_20180823T152254.SAFE.log
2019-03-25 09:18:50,255 - Start-Maja - INFO - #################################
2019-03-25 09:51:25,266 - Start-Maja - INFO - => processing date 20180828
2019-03-25 09:51:25,269 - Start-Maja - INFO - Using MUSCATE L2 type
2019-03-25 09:51:25,269 - Start-Maja - INFO - previous L2 : /mnt/passesData/MAJAConfFold/L2A_MAJA/Spain/29TPH/GIPP_MAJA_3.1.2_TM/SENTINEL2B_20180823-112334-374_L2A_T29TPH_C_V1-0
2019-03-25 09:51:25,276 - Start-Maja - INFO - #################################
2019-03-25 09:51:25,276 - Start-Maja - INFO - #################################
2019-03-25 09:51:25,276 - Start-Maja - INFO - processing /mnt/passesData/MAJAConfFold/L1C_PDGS/Spain/S2A_MSIL1C_20180828T112111_N0206_R037_T29TPH_20180828T135154.SAFE in nominal mode
2019-03-25 09:51:25,276 - Start-Maja - INFO - MAJA logfile: /mnt/passesData/MAJAConfFold/L2A_MAJA/Spain/29TPH/GIPP_MAJA_3.1.2_TM//S2A_MSIL1C_20180828T112111_N0206_R037_T29TPH_20180828T135154.SAFE.log
2019-03-25 09:51:25,276 - Start-Maja - INFO - #################################
2019-03-25 10:12:19,076 - Start-Maja - INFO - => processing date 20180902
2019-03-25 10:12:19,079 - Start-Maja - INFO - Using MUSCATE L2 type
2019-03-25 10:12:19,080 - Start-Maja - INFO - previous L2 : /mnt/passesData/MAJAConfFold/L2A_MAJA/Spain/29TPH/GIPP_MAJA_3.1.2_TM/SENTINEL2A_20180828-112246-266_L2A_T29TPH_C_V1-0
2019-03-25 10:12:19,086 - Start-Maja - INFO - #################################
2019-03-25 10:12:19,086 - Start-Maja - INFO - #################################
2019-03-25 10:12:19,086 - Start-Maja - INFO - processing /mnt/passesData/MAJAConfFold/L1C_PDGS/Spain/S2B_MSIL1C_20180902T112109_N0206_R037_T29TPH_20180902T140408.SAFE in nominal mode
2019-03-25 10:12:19,086 - Start-Maja - INFO - MAJA logfile: /mnt/passesData/MAJAConfFold/L2A_MAJA/Spain/29TPH/GIPP_MAJA_3.1.2_TM//S2B_MSIL1C_20180902T112109_N0206_R037_T29TPH_20180902T140408.SAFE.log
2019-03-25 10:12:19,086 - Start-Maja - INFO - #################################
And in repL2 folder:
S2A_MSIL1C_20180828T112111_N0206_R037_T29TPH_20180828T135154.SAFE.log
S2A_OPER_PMC_L2REPT_29TPH____20180828.EEF
S2A_TEST_QCK_L2VALD_T29TPH_20180828.DBL.DIR
S2A_TEST_QCK_L2VALD_T29TPH_20180828.HDR
S2B_MSIL1C_20180823T112109_N0206_R037_T29TPH_20180823T152254.SAFE.log
S2B_MSIL1C_20180902T112109_N0206_R037_T29TPH_20180902T140408.SAFE.log
S2B_OPER_PMC_L2REPT_29TPH____20180823.EEF
S2B_OPER_PMC_L2REPT_29TPH____20180902.EEF
S2B_TEST_QCK_L2VALD_T29TPH_20180823.DBL.DIR
S2B_TEST_QCK_L2VALD_T29TPH_20180823.HDR
S2B_TEST_QCK_L2VALD_T29TPH_20180902.DBL.DIR
S2B_TEST_QCK_L2VALD_T29TPH_20180902.HDR
SENTINEL2A_20180828-112246-266_L2A_T29TPH_C_V1-0
SENTINEL2B_20180823-112334-374_L2A_T29TPH_C_V1-0
SENTINEL2B_20180902-112716-061_L2A_T29TPH_C_V1-0
As you can see I have got each .SAFE.log (not empty) and the L2A of all scenes processed. In my first attempts I was only getting the L2A product of one of them.
Regards
from maja.
olivierhagolle
commented on June 11, 2024
That's good news !
Sorry to have taken so long figuring it out.
Olivier
from maja.
Related Issues (20)
- How to install MAJA 4 in Ubuntu 20.04 and Docker HOT 3
- [best practices] Processing by orbit HOT 11
- Using StartMaja with MAJA v3.2.2 HOT 1
- Solution for the error happening in case you try to use CAMS (if "NbNonInterpolate" in self._beforeFile.keys(): 'NoneType' object has no attribute 'keys') HOT 3
- Testing MAJA - Where to get "MAJA-DATA" and "MAJA-Validation"? HOT 1
- Github vs gitlab - which MAJA code should I use? HOT 3
- What are the differences between StartMaja scripts on different repositories? HOT 1
- License confusion HOT 2
- Access denied HOT 11
- Xml encoding in xml_tools.py HOT 2
- SRTM download link doesn't respond HOT 5
- Optimal parameters to get the best results (advice needed) HOT 10
- Inputs do not occupy the same physical space HOT 5
- CMake error for EXPAT when compiling MAJA HOT 11
- [best practices] Optimal parameters for rain forest region HOT 4
- Cmake error for GDAL while compiling MAJA
- SameFileError when processing with StartMaja HOT 1
- Repository no more active, please use MAJA's Forum for issues
- Directory of input files as symbolic links
- Maja creates L2A level data with empty cloud masks & 0 instead of -10000 for each band HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from maja.