Comments (21)
Hi Ruben,
sorry for late answer, i had not noticed this issue.
I used the following command (with tiles.kml being the file provided by ESA which contains all the tiles definitions). I had already downloaded all the SWBD tiles in the /work/CESBIO/projects/Maja/SWBD directory, and this includes all the SWBD tiles necessary:
python DTMCreation.py -k tiles.kml -g 49MFT -s /work/CESBIO/projects/Maja/SRTM -w /work/CESBIO/projects/Maja/SWBD -o ../DTM
As for you, the e111s02 tile is missing, but I get
('missing SWBD watr file : ', 'e111s02')
it is a fully land tile
/work/CESBIO/projects/Maja/SWBD/e111s02.gml
('#############Fichier eau :', '/work/CESBIO/projects/Maja/SWBD/e111s02.gml')
gdal_rasterize -burn 0 -l e111s02 /work/CESBIO/projects/Maja/SWBD/e111s02.gml /tmp/49MFTS5IW8b/tmpjO7ymF/srtm_59_13_tmp.tif
Warning 1: The output raster dataset has a SRS, but the input vector layer SRS is unknown.
Ensure input vector has the same SRS, otherwise results might be incorrect.
0...10...20...30...40...50...60...70...80...90...100 - done.
It worked with no issue.
Did you download the latest version of Start-Maja ?
You should have this directory in your installation:
https://github.com/CNES/Start-MAJA/tree/master/prepare_dtm/land_polygons_osm
Can you check it ?
Olivier
from maja.
Hi Olivier,
Thank you very much for checking it.
I have run again DTMCreation.py
and it worked properly.
e111s02
('missing SWBD watr file : ', 'e111s02')
it is a fully land tile
/mnt/passesData/Ruben/DTMsTest/DTM/Water//e111s02.gml
('#############Fichier eau :', '/mnt/passesData/Ruben/DTMsTest/DTM/Water//e111s02.gml')
gdal_rasterize -burn 0 -l e111s02 /mnt/passesData/Ruben/DTMsTest/DTM/Water//e111s02.gml /tmp/49MFTuyNuP7/tmpITdYoh/srtm_59_13_tmp.tif
Warning 1: The output raster dataset has a SRS, but the input vector layer SRS is unknown.
Ensure input vector has the same SRS, otherwise results might be incorrect.
0...10...20...30...40...50...60...70...80...90...100 - done.
I believe I know where the issue were coming from.
I am curious how MAJA works here. As you can see in the screenshot there is a small area in red which is not covered by the tiles and that one should be e111s02. However, when I opened the output product the whole tile was processed. Is MAJA doing some kind of extrapolation to fill in that area with data?
Ruben
from maja.
Sorry,
I forgot to attach the file.
I would like to ask something else that I forgot before. MAJA uses the SRTM DEM, however, it is not covering latitudes above 60 degrees so if I want to run it in the north of Canada, could I use ASTER DEM?
Thanks
Ruben
from maja.
Hi, thanks for the good news.
yes, SWBD tiles are not provided when they are fully land or fully water. The only issue is to know which of these two cases is to be considered, and that's easy to know, using a data base, or the DTM. So DTMCreation, makes a fully land or water tile, and then processes it as if it were a normal SWBD file.
Nothing prevents from using another DEM, but we did not prepare the tool to convert it. It is among the things to do... If you want to contribute an adaptation of DTMCreation to use AsterDem, you are very welcome ;)
Best regards,
Olivier
from maja.
Hi,
I have exactly the same error for a particular tile.
I downloaded the right versions of Maja, Start_Maja etc... and i have all SRTM and SWBD archives stored in a directory.
I successfully ran MAJA on another tile so the installation is OK.
python` /data/MAJA_to_Run/Start-MAJA/prepare_dtm/DTMCreation.py -k /data/MAJA_to_Run/TempFile_MAJA/kml_grid_tiles/S2A_OPER_GIP_TILPAR_MPC__20151209T095117_V20150622T000000_21000101T000000_B00.kml -g 28QCD -s /data/MAJA_to_Run/srtm_water_forMAja/srtm -w /data/MAJA_to_Run/srtm_water_forMAja/swbd -o /data/MAJA_to_Run/Start-MAJA/DTM/28QCD
Found SRTM zip-archives...
Found Water zip-archives...
Processing with GranuleID
28QCD
UTM28N
32628
EPSG:32628
0
0
0
0
109800
109800
0
300000
1900020
Working directory: /tmp/28QCDOrRUJ_
('/data/MAJA_to_Run/srtm_water_forMAja/srtm', '/tmp/28QCDOrRUJ_', '/data/MAJA_to_Run/srtm_water_forMAja/swbd', '/tmp/28QCDOrRUJ_')
[33, 9]
[33, 9]
((-16.88042164508539, 17.176229136834404, 0.0), (-15.84383635471145, 16.19074562988601, 0.0))
([33, 9], [33, 9])
['srtm_33_09.tif']
((-16.88042164508539, 17.176229136834404, 0.0), (-15.84383635471145, 16.19074562988601, 0.0))
([-17, 17], [-16, 16])
('longitudes', -17, -16)
('latitudes', 16, 17)
('center coordinates', [[-16.5, 16.5], [-16.5, 17.5], [-15.5, 16.5], [-15.5, 17.5]])
['w017n16', 'w017n17', 'w016n16', 'w016n17']
/tmp/28QCDOrRUJ_
('liste_fic_mnt', ['srtm_33_09.tif'])
Archive: /data/MAJA_to_Run/srtm_water_forMAja/swbd/w017n16f.zip
extracting: /tmp/28QCDOrRUJ_/tmpr0O_8f/w017n16f.shp
extracting: /tmp/28QCDOrRUJ_/tmpr0O_8f/w017n16f.dbf
extracting: /tmp/28QCDOrRUJ_/tmpr0O_8f/w017n16f.shx
Archive: /data/MAJA_to_Run/srtm_water_forMAja/swbd/w017n17f.zip
extracting: /tmp/28QCDOrRUJ_/tmpr0O_8f/w017n17f.shp
extracting: /tmp/28QCDOrRUJ_/tmpr0O_8f/w017n17f.dbf
extracting: /tmp/28QCDOrRUJ_/tmpr0O_8f/w017n17f.shx
Archive: /data/MAJA_to_Run/srtm_water_forMAja/swbd/w016n16f.zip
extracting: /tmp/28QCDOrRUJ_/tmpr0O_8f/w016n16f.shp
extracting: /tmp/28QCDOrRUJ_/tmpr0O_8f/w016n16f.dbf
extracting: /tmp/28QCDOrRUJ_/tmpr0O_8f/w016n16f.shx
FIC: srtm_33_09.tif
(<type 'str'>, <type 'str'>)
/data/MAJA_to_Run/srtm_water_forMAja/srtm/srtm_33_09.tif
Archive: /data/MAJA_to_Run/srtm_water_forMAja/srtm/srtm_33_09.zip
extracting: /tmp/28QCDOrRUJ_/tmpr0O_8f/srtm_33_09.tif
gdalwarp -r cubic -srcnodata -32767 -dstnodata 0 /tmp/28QCDOrRUJ_/tmpr0O_8f/srtm_33_09.tif /tmp/28QCDOrRUJ_/tmpr0O_8f/srtm_33_09nodata0.tif
Creating output file that is 6000P x 6000L.
Processing /tmp/28QCDOrRUJ_/tmpr0O_8f/srtm_33_09.tif [1/1] : 0...10...20...30...40...50...60...70...80...90...100 - done.
w017n16
('#############Fichier eau :', '/tmp/28QCDOrRUJ_/tmpr0O_8f/w017n16f.shp')
gdal_rasterize -burn 1 -l w017n16f /tmp/28QCDOrRUJ_/tmpr0O_8f/w017n16f.shp /tmp/28QCDOrRUJ_/tmpr0O_8f/srtm_33_09_tmp.tif
Warning 1: The output raster dataset has a SRS, but the input vector layer SRS is unknown.
Ensure input vector has the same SRS, otherwise results might be incorrect.
0...10...20...30...40...50...60...70...80...90...100 - done.
w017n17
('#############Fichier eau :', '/tmp/28QCDOrRUJ_/tmpr0O_8f/w017n17f.shp')
gdal_rasterize -burn 1 -l w017n17f /tmp/28QCDOrRUJ_/tmpr0O_8f/w017n17f.shp /tmp/28QCDOrRUJ_/tmpr0O_8f/srtm_33_09_tmp.tif
Warning 1: The output raster dataset has a SRS, but the input vector layer SRS is unknown.
Ensure input vector has the same SRS, otherwise results might be incorrect.
0...10...20...30...40...50...60...70...80...90...100 - done.
w016n16
('#############Fichier eau :', '/tmp/28QCDOrRUJ_/tmpr0O_8f/w016n16f.shp')
gdal_rasterize -burn 1 -l w016n16f /tmp/28QCDOrRUJ_/tmpr0O_8f/w016n16f.shp /tmp/28QCDOrRUJ_/tmpr0O_8f/srtm_33_09_tmp.tif
Warning 1: The output raster dataset has a SRS, but the input vector layer SRS is unknown.
Ensure input vector has the same SRS, otherwise results might be incorrect.
0...10...20...30...40...50...60...70...80...90...100 - done.
w016n17
('missing SWBD watr file : ', 'w016n17')
Traceback (most recent call last):
File "/data/MAJA_to_Run/Start-MAJA/prepare_dtm/DTMCreation.py", line 326, in <module>
creator.run(args.out, args.tempout)
File "/data/MAJA_to_Run/Start-MAJA/prepare_dtm/DTMCreation.py", line 286, in run
water_zipped = self.WaterZipped)
File "/data/MAJA_to_Run/Start-MAJA/prepare_dtm/../prepare_dtm/tuilage_mnt_eau_S2.py", line 203, in run
calcul_masque_eau_mnt, working_dir=working_dir)
File "/data/MAJA_to_Run/Start-MAJA/prepare_dtm/../prepare_dtm/lib_mnt.py", line 524, in fusion_mnt
land = TestLand(liste_centre_eau[i][0], liste_centre_eau[i][1])
File "/data/MAJA_to_Run/Start-MAJA/prepare_dtm/../prepare_dtm/lib_mnt.py", line 57, in TestLand
layer = dataSource.GetLayer()
AttributeError: 'NoneType' object has no attribute 'GetLayer'
thanks for your help
from maja.
Hi Resnow,
Sorry for replying late.
Can you please check that you have this directory in your installation ?
https://github.com/CNES/Start-MAJA/tree/master/prepare_dtm/land_polygons_osm
Olivier
from maja.
Hi, thanks for your response.
Yes I have the directory you mentioned.
from maja.
Hi,
OK, I think I know where the error comes from.
I run prepare_dtm from the prepare_dtm folder. There is a relative path in the code, that should be fixed.
shapefile = "land_polygons_osm/simplified_land_polygons.shp"
As a workaround until I correct it, please run the code from the prepare_dtm folder.
Olivier
from maja.
Many thanks it is working.
For information I also noticed that I need to move the DTM files freshly created in the GIPP directory. Otherwise I have an error saying that DTM doesn't exist.
For the moment I move manually the files.
from maja.
Good news, I will fix that as soon as I get 30 clear minutes.
The DTM should be stored in a DTM/ folder within start_maja folder.
Olivier
from maja.
Yes I know that DTM have to be stored in Start_Maja/DTM/ but it is not working.
It is working only when i move the dtm files in GIPP directory.
from maja.
This is strange. Would you have an example of command, folders.txt and error message ?
Olivier
from maja.
the command for dtm is:
python /data/MAJA_to_Run/Start-MAJA/prepare_dtm/DTMCreation.py -k /data/MAJA_to_Run/TempFile_MAJA/kml_grid_tiles/S2A_OPER_GIP_TILPAR_MPC__20151209T095117_V20150622T000000_21000101T000000_B00.kml -g 28QCD -s /data/MAJA_to_Run/srtm_water_forMAja/srtm -w /data/MAJA_to_Run/srtm_water_forMAja/swbd -o /data/MAJA_to_Run/Start-MAJA/DTM/28QCD
the commands for Maja is :
python /data/MAJA_to_Run/Start-MAJA/start_maja.py -f /data/MAJA_to_Run/Start-MAJA/folders.txt -g GIPP_S2_MAJA_3.3_TM -l LUT_MAJA_3_TM_CAMS -t 28QCD -s TEST -d 20170101 -e 20171231
attached capture of error message and folder.txt
I have this error when dtm files are by defeult in /FTM folder. But Maja working well when i move these files in GIPP directory.
from maja.
Thanks for all this, I guess I have all what is needed to search for the error.
By the way, I see that you are using only three images in backward mode, our recommendation is 8 (But maybe was it only for test purposes and you know that already). And all the images should be included, including the cloudy ones.
The more frequent the images, the better results.
Olivier
from maja.
Even image completly cloudy ? Because I download images with a cloud cover threshold depending of the application.
Thanks for your help and to take a moment to have a look on the error mentioned.
from maja.
Images completely cloudy are not necessary, but the L1C cloud cover rate is not very accurate. I find it easier to download everything and let MAJA decide what is a cloud.
from maja.
Ok thanks for your advice.
from maja.
Hello Olivier,
I am having the same problem I had a couple of months ago. In this case, I am running Start-MAJA
within a Dockerfile and I am getting the error with the missing file.
python /home/gaiascope/Start-MAJA/prepare_dtm/DTMCreation.py -k /home/gaiascope/S2A_OPER_GIP_TILPAR_MPC__20151209T095117_V20150622T000000_21000101T000000_B00.kml -g 31UCU -s /home/gaiascope/DTM_SWBD/Creation/SRTM -w /home/gaiascope/DTM_SWBD/Creation/Water -o /out/DTM_SWBD/DTMOut
Archive: /home/gaiascope/DTM_SWBD/Creation/Water/e000n52e.zip
inflating: /tmp/31UCUIJtfUS/tmppEF9O7/e000n52e.dbf
inflating: /tmp/31UCUIJtfUS/tmppEF9O7/e000n52e.shp
inflating: /tmp/31UCUIJtfUS/tmppEF9O7/e000n52e.shx
creating: /tmp/31UCUIJtfUS/tmppEF9O7/Documents/
inflating: /tmp/31UCUIJtfUS/tmppEF9O7/Documents/Readme File SRTM Water Body Data.doc
inflating: /tmp/31UCUIJtfUS/tmppEF9O7/Documents/SRTM Edit Rules v2.0 12 Mar 03.doc
inflating: /tmp/31UCUIJtfUS/tmppEF9O7/Documents/SWBD Product Specific Guidance v2.0 12 Mar 03jas.doc
Archive: /home/gaiascope/DTM_SWBD/Creation/Water/e000n53e.zip
inflating: /tmp/31UCUIJtfUS/tmppEF9O7/e000n53e.dbf
inflating: /tmp/31UCUIJtfUS/tmppEF9O7/e000n53e.shp
inflating: /tmp/31UCUIJtfUS/tmppEF9O7/e000n53e.shx
inflating: /tmp/31UCUIJtfUS/tmppEF9O7/Documents/Readme File SRTM Water Body Data.doc
inflating: /tmp/31UCUIJtfUS/tmppEF9O7/Documents/SRTM Edit Rules v2.0 12 Mar 03.doc
inflating: /tmp/31UCUIJtfUS/tmppEF9O7/Documents/SWBD Product Specific Guidance v2.0 12 Mar 03jas.doc
Archive: /home/gaiascope/DTM_SWBD/Creation/Water/e001n52e.zip
inflating: /tmp/31UCUIJtfUS/tmppEF9O7/e001n52e.dbf
inflating: /tmp/31UCUIJtfUS/tmppEF9O7/e001n52e.shp
inflating: /tmp/31UCUIJtfUS/tmppEF9O7/e001n52e.shx
inflating: /tmp/31UCUIJtfUS/tmppEF9O7/Documents/Readme File SRTM Water Body Data.doc
inflating: /tmp/31UCUIJtfUS/tmppEF9O7/Documents/SRTM Edit Rules v2.0 12 Mar 03.doc
inflating: /tmp/31UCUIJtfUS/tmppEF9O7/Documents/SWBD Product Specific Guidance v2.0 12 Mar 03jas.doc
Archive: /home/gaiascope/DTM_SWBD/Creation/SRTM/srtm_37_02.zip
inflating: /tmp/31UCUIJtfUS/tmppEF9O7/readme.txt
inflating: /tmp/31UCUIJtfUS/tmppEF9O7/srtm_37_02.hdr
inflating: /tmp/31UCUIJtfUS/tmppEF9O7/srtm_37_02.tfw
inflating: /tmp/31UCUIJtfUS/tmppEF9O7/srtm_37_02.tif
Creating output file that is 6001P x 6001L.
Processing input file /tmp/31UCUIJtfUS/tmppEF9O7/srtm_37_02.tif.
0...10...20...30...40...50...60...70...80...90...100 - done.
0Warning 1: The output raster dataset has a SRS, but the input vector layer SRS is unknown.
Ensure input vector has the same SRS, otherwise results might be incorrect.
...10...20...30...40...50...60...70...80...90...100 - done.
0Warning 1: The output raster dataset has a SRS, but the input vector layer SRS is unknown.
Ensure input vector has the same SRS, otherwise results might be incorrect.
...10...20...30...40...50...60...70...80...90...100 - done.
Warning 1: The output raster dataset has a SRS, but the input vector layer SRS is unknown.
Ensure input vector has the same SRS, otherwise results might be incorrect.
0...10...20...30...40...50...60...70...80...90...100 - done.
Traceback (most recent call last):
File "/home/gaiascope/Start-MAJA/prepare_dtm/DTMCreation.py", line 326, in <module>
creator.run(args.out, args.tempout)
File "/home/gaiascope/Start-MAJA/prepare_dtm/DTMCreation.py", line 286, in run
Found SRTM zip-archives...
Found Water zip-archives...
Processing with GranuleID
31UCU
UTM31N
32631
EPSG:32631
0
0
0
0
109800
109800
0
300000
5900040
Working directory: /tmp/31UCUIJtfUS
('/home/gaiascope/DTM_SWBD/Creation/SRTM', '/tmp/31UCUIJtfUS', '/home/gaiascope/DTM_SWBD/Creation/Water', '/tmp/31UCUIJtfUS')
[37, 2]
[37, 2]
((0.00476092508203617, 53.211977826146494, 0.0), (1.678544065017885, 52.255150748300906, 0.0))
([37, 2], [37, 2])
['srtm_37_02.tif']
((0.00476092508203617, 53.211977826146494, 0.0), (1.678544065017885, 52.255150748300906, 0.0))
([0, 53], [1, 52])
('longitudes', 0, 1)
('latitudes', 52, 53)
('center coordinates', [[0.5, 52.5], [0.5, 53.5], [1.5, 52.5], [1.5, 53.5]])
['e000n52', 'e000n53', 'e001n52', 'e001n53']
/tmp/31UCUIJtfUS
('liste_fic_mnt', ['srtm_37_02.tif'])
FIC: srtm_37_02.tif
(<type 'str'>, <type 'str'>)
/home/gaiascope/DTM_SWBD/Creation/SRTM/srtm_37_02.tif
gdalwarp -r cubic -srcnodata -32767 -dstnodata 0 /tmp/31UCUIJtfUS/tmppEF9O7/srtm_37_02.tif /tmp/31UCUIJtfUS/tmppEF9O7/srtm_37_02nodata0.tif
e000n52
('#############Fichier eau :', '/tmp/31UCUIJtfUS/tmppEF9O7/e000n52e.shp')
gdal_rasterize -burn 1 -l e000n52e /tmp/31UCUIJtfUS/tmppEF9O7/e000n52e.shp /tmp/31UCUIJtfUS/tmppEF9O7/srtm_37_02_tmp.tif
e000n53
('#############Fichier eau :', '/tmp/31UCUIJtfUS/tmppEF9O7/e000n53e.shp')
gdal_rasterize -burn 1 -l e000n53e /tmp/31UCUIJtfUS/tmppEF9O7/e000n53e.shp /tmp/31UCUIJtfUS/tmppEF9O7/srtm_37_02_tmp.tif
e001n52
('#############Fichier eau :', '/tmp/31UCUIJtfUS/tmppEF9O7/e001n52e.shp')
gdal_rasterize -burn 1 -l e001n52e /tmp/31UCUIJtfUS/tmppEF9O7/e001n52e.shp /tmp/31UCUIJtfUS/tmppEF9O7/srtm_37_02_tmp.tif
e001n53
('missing SWBD watr file : ', 'e001n53')
water_zipped = self.WaterZipped)
File "/home/gaiascope/Start-MAJA/prepare_dtm/../prepare_dtm/tuilage_mnt_eau_S2.py", line 203, in run
calcul_masque_eau_mnt, working_dir=working_dir)
File "/home/gaiascope/Start-MAJA/prepare_dtm/../prepare_dtm/lib_mnt.py", line 524, in fusion_mnt
land = TestLand(liste_centre_eau[i][0], liste_centre_eau[i][1])
File "/home/gaiascope/Start-MAJA/prepare_dtm/../prepare_dtm/lib_mnt.py", line 57, in TestLand
layer = dataSource.GetLayer()
AttributeError: 'NoneType' object has no attribute 'GetLayer'
1
The area I am covering is close to the coast (see screenshot):
I am wondering if you have run MAJA with Dockerfiles, if so, do you know why this is happening?
Thanks
Ruben
from maja.
Hi Ruben,
DTMCreation.py is not well written, and some paths to find auxiliary data to fill the gaps in SWBD are provided in relative to where the execution is launched. I should correct that but did not have time so far.
I think you would get a better result with:
cd /home/gaiascope/Start-MAJA/prepare_dtm/
DTMCreation.py -k /home/gaiascope/S2A_....
Best regards,
Olivier
from maja.
Hi Olivier,
Thank you very much.
I changed to cd and then run the python script as per your suggestion and it worked.
Regards
Ruben
from maja.
Good news !
from maja.
Related Issues (20)
- How to install MAJA 4 in Ubuntu 20.04 and Docker HOT 3
- [best practices] Processing by orbit HOT 11
- Using StartMaja with MAJA v3.2.2 HOT 1
- Solution for the error happening in case you try to use CAMS (if "NbNonInterpolate" in self._beforeFile.keys(): 'NoneType' object has no attribute 'keys') HOT 3
- Testing MAJA - Where to get "MAJA-DATA" and "MAJA-Validation"? HOT 1
- Github vs gitlab - which MAJA code should I use? HOT 3
- What are the differences between StartMaja scripts on different repositories? HOT 1
- License confusion HOT 2
- Access denied HOT 11
- Xml encoding in xml_tools.py HOT 2
- SRTM download link doesn't respond HOT 5
- Optimal parameters to get the best results (advice needed) HOT 10
- Inputs do not occupy the same physical space HOT 5
- CMake error for EXPAT when compiling MAJA HOT 11
- [best practices] Optimal parameters for rain forest region HOT 4
- Cmake error for GDAL while compiling MAJA
- SameFileError when processing with StartMaja HOT 1
- Repository no more active, please use MAJA's Forum for issues
- Directory of input files as symbolic links
- Maja creates L2A level data with empty cloud masks & 0 instead of -10000 for each band HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from maja.