genericmappingtools / remote-datasets Goto Github PK

As it was suggested in #92, we should use thumbnails for the front page ( https://www.generic-mapping-tools.org/remote-datasets/index.html)

For a two column content, we could use images with 400px wide.
Should we use three-column instead?

I think I can make maps without legends/colorbar. We could leave that for each individual page.

For the images, I am thinking of using the same name and add the "thumbnail" suffix.

Any comment, idea or suggestion?

There are some small differences in the static figures (https://www.generic-mapping-tools.org/remote-datasets/index.html).

The idea is to make all similar (and add scripts to those that don't have).

Important doubts:

1. Should I extend the maps at to the poles (for those that don't have data their)? I think we should so the users can clearly see the lack of the data there (and also all the figures have the same size).

Minor doubts:

1. With or without a frame? Some of them have.
2. There are maps with jpg and png format. I should respect that, right?

This is mainly a recordatory:

We should update the webpage for the:

• New grids: moon and other planets
• Updated grids: update the information.

I am thinking of adding the grid's version in the title (in this page). I would like to easly see what version are available.

Hi every body,

I want to focus your attention on these two databases for your consideration. I think that will be very useful to have direct access to them. I share with you the links to the main projects of both of them:

• GEBCO
• GSHHS

Indeed I code some workaround, I currently use to extract GSHHS coastline (shpfiles) using gepandas, you can check the feature request I did in pyGMT . I think these features will be easy to implement and very useful for the community.

Have a nice weekend!

Now that there are many datasets, I think it could be useful to have a way to call the maximum resolution available. This way the user will have no need to check in the docs which is the highest.

Maybe something like @earth_mag_max.

Right now, pieces like earth-faa.rst are manually maintained. However, with soon > 20 datasets this gets tedious. I suggest we instead maintain either a single text file with the background on each data set (or, alternatively, one small txt file for each). Then, a script can loop over these and make individual RST snippet like earth-faa.rst. In addition to the preamble given by the small txt files and the inclusion of files like GMT_faa.jpg), the script will need to

• Format the Usage section
• Build the tables (e.g., _tbl-earth_faa). The dimensions and etc is available on the candidate server, for instance, or locally.
• Any data references might be comments in the txt file that if > 0 records builds a Data References section

For many purposes it would be nice to somehow get, say, earth_relief_15s.grd downloaded in one piece instead of having GMT reassemble all the tiles. It is particularly useful if one is running animations and needs a global grid. Unless it is 6m and up we have to run grdcut to assemble the higher dimension global file (which we made during the processing).

Once option to implement this would be to simply distinguish between @earth_relief_15s[_p|g] and @earth_relief_15s[_p|g].grd, the latter requesting the whole enchilada. These would then have to be uploaded, of course, and placed in the gmt_data_server.txt in a similar way to the lower wool-earth grids.

Negative is this duplicates memory requirements for the biggest data sets. Any thoughts?

In PR #16, the sphinx-panels extension is used to improve the landing page, but the sphinx-panels extension will be deprecated and be replaced by sphinx-design (executablebooks/sphinx-panels#67).

The syntax of the sphinx-design extension is also easier to read and understand. We should migrate sphinx-panels to sphinx-design.

I'd like to help with the migration if agreed.

It seems that in June a new version was released.

https://www.gebco.net/data_and_products/gridded_bathymetry_data/gebco_2022/

This issue is for tracking progress on the announcements of the new datasets.

• Create animation or image showing off new datasets
• Publish Forum announcement
• Post on twitter
• Post on instagram
• Post on website news section

As suggested in #4, this feature request is to add a simple example or two for each remote dataset, similar to those in the GMT man pages.

After PR #101 there are now both PNG and JPG files stored in the folder at https://github.com/GenericMappingTools/remote-datasets/tree/main/docs/_static for the GMT_earth_age and the GMT_earth_mask remote dataset images. For the thumbnail images on the overview page, the JPG files are used, but for the large images on the separate documentation pages, the PNG files are used. For GMT_earth_mask the PNG and JPG files are not identical (please compare https://www.generic-mapping-tools.org/remote-datasets/ and https://www.generic-mapping-tools.org/remote-datasets/earth-mask.html).
Maybe it makes sense to consistently use one format, i.e., the JPG format, across all remote datasets and delete the PNG files?

The file sizes in this table are wrong.

For example it says that the size for the 30s image is 935 MB when in fact is 219M (394 M) for the day (nigth) image. And the sizes (based on the info of gmt_data_server.txt) are consirable different for most of the resolutions.

What should I do?

1. Only use the sizes for day image?
2. Only use the sizes for night image?
3. Use both sizes in the same table?
4. Make another table?

Description of the desired feature

Currently, there is no page that lists all remote files, what they are, where they come from, and what they contain. We should have such a page. Bonus points if there are plots these files as well.

https://www.generic-mapping-tools.org/remote-datasets/earth-masks.html

The documentation for earth-mask does not explain what the values mean in the earth-mask grids, although I can guess that 0 means ocean and 1 means land from the legend of the top figure.

I think we should:

• Clearly explain the meaning of the values 0-4
• Give the command that is used to generate the earth_mask grids (e.g., grdlandmask -N0/1/2/3/4)

In the webpage it says that the registration of the geoid_01m is p but in the server it is g.

The info on the webpage is wrong, right?

https://www.generic-mapping-tools.org/remote-datasets/earth-geoid.html
https://oceania.generic-mapping-tools.org/server/earth/earth_geoid/

It seems that David will release a new version soon.
I think I could handle it.

Since march 6th there is a new version (2.4) of SRTM15+ (@earth_relief).
https://topex.ucsd.edu/pub/srtm15_plus/

Hi @Esteban82-

Could you give me an update on where we are with

1. The candidate server (I think it is fully populated, no?)
2. Illustrations for each data set to go on the website showing the data
3. Any scripts that still need to be made?

I think we need to make sure that this repo is completed, and then when we actually release GMT 6.5 then we can run

make server-release

Which we have ever done (we have dome make candidate-release) but have to look out for issues live.n

It seems there is an error in the processing of @earth_relief_30s_g.

gmt begin earth_relief png
	gmt grdimage @earth_relief_30s_g -Bf -Cgeo -JM5c -B+t"relief_30s_g" -R-72/-64/-35/-30
	gmt grdimage @earth_relief_30s_p -Bf -Cgeo -X5c -B+t"relief_30s_p"
	gmt grdimage @earth_relief_01m_g -Bf -Cgeo -X5c -B+t"relief_01m_g"
	gmt grdimage @earth_gebco_30s_g -Bf -Cgeo -X5c -B+t"gebco_30s_g"
gmt end

Pinging @GenericMappingTools/gmt-team. I have uploaded new remote data grids to the test server. These are described here. Note that until we complete the dvc move for test images we have not yet updated the earth_relief files (they are still based off SRTM15+ v2.1 from 2019). Once that is settled I will update those files to 2.3 and add the synbath grid as well. Because of this, if you run all the GMT tests they still all pass since all the DEM images are made using the same DEM as was used in the originals (once we update the DEM we will have ~60 failures...).

In the meantime, I hope you can find time to play a bit with the new datasets. To do so, and not mess up your other remote files, I suggest you do this:

Edit your ConfigUserAdvanced.cmake file (perhaps after updating via the Template) and uncomment this line:

# set (GMT_DATA_SERVER "test")

then rebuild GMT. To preserve your existing cache and server files, simply set them aside via

mv ~/.gmt/server ~/.gmt/server_orig
mv ~/.gmt/cache ~/.gmt/cache_orig

so that you will only use files from the test server. You can now try out making maps with @earth_{gebco,gebcosi,geoid,mag,mag4km,vgg,wdmam} - they all have their own default CPTs that will be downloaded via the cache.

Let me know if you find anything odd. The default CPTs are the ones recommended by the data provider, except Gebco which has not replied. I am therefore using the same geo CPT for all the reliefs.

Once you have finished testing, remove the new directories and rename the old:

rm -f /.gmt/server  ~/.gmt/cache 
mv ~/.gmt/server_orig ~/.gmt/server 
mv ~/.gmt/cache_orig ~/.gmt/cache

Description of the problem

Hello guys! When I use grid = pygmt.datasets.load_earth_relief(resolution="06m", region=[70, 140, 0, 55]) to set a map of map, I found the provided resolution code can not be used.

Minimal Complete Verifiable Example

For example, when I set the resolution as "03m*", as codes as :

grid = pygmt.datasets.load_earth_relief(resolution="03m*", region=[70, 140, 0, 55])
fig = pygmt.Figure()
fig.grdimage(grid=grid, projection="M15c", frame="a", cmap="geo")
fig.colorbar(frame=["a1000", "x+lElevation", "y+lm"])
fig.show()

Full error message

---------------------------------------------------------------------------
GMTInvalidInput                           Traceback (most recent call last)
Cell In[21], line 1
----> 1 grid = pygmt.datasets.load_earth_relief(resolution="03m*", region=[70, 140, 0, 55])
      2 fig = pygmt.Figure()
      3 fig.grdimage(grid=grid, projection="M15c", frame="a", cmap="geo")

File f:\Anaconda3\envs\GeoTable\lib\site-packages\pygmt\helpers\decorators.py:738, in kwargs_to_strings..converter..new_module(*args, **kwargs)
    736             kwargs[arg] = separators[fmt].join(f"{item}" for item in value)
    737 # Execute the original function and return its output
--> 738 return module_func(*args, **kwargs)

File f:\Anaconda3\envs\GeoTable\lib\site-packages\pygmt\datasets\earth_relief.py:159, in load_earth_relief(resolution, region, registration, data_source, use_srtm)
    156 else:
    157     dataset_prefix = earth_relief_sources[data_source]
--> 159 grid = _load_remote_dataset(
    160     dataset_name="earth_relief",
    161     dataset_prefix=dataset_prefix,
    162     resolution=resolution,
    163     region=region,
    164     registration=registration,
    165 )
    166 return grid

File f:\Anaconda3\envs\GeoTable\lib\site-packages\pygmt\helpers\decorators.py:738, in kwargs_to_strings..converter..new_module(*args, **kwargs)
...
    279 # check registration
    280 if registration is None:
    281     # use gridline registration unless only pixel registration is available

GMTInvalidInput: Invalid resolution '03m*'.

System information

PyGMT information:
  version: v0.10.0
System information:
  python: 3.10.13 | packaged by Anaconda, Inc. | (main, Sep 11 2023, 13:24:38) [MSC v.1916 64 bit (AMD64)]
  executable: f:\Anaconda3\envs\GeoTable\python.exe
  machine: Windows-10-10.0.19041-SP0
Dependency information:
  numpy: 1.26.1
  pandas: 2.1.1
  xarray: 2023.10.1
  netCDF4: 1.6.5
  packaging: 23.2
  contextily: None
  geopandas: None
  IPython: 8.17.2
  rioxarray: None
  ghostscript: 10.02.0
GMT library information:
  binary version: 6.4.0
  cores: 12
  grid layout: rows
  image layout: 
  library path: F:/Anaconda3/envs/GeoTable/Library/bin/gmt.dll
  padding: 2
  plugin dir: F:/Anaconda3/envs/GeoTable/Library/bin/gmt_plugins
  share dir: F:/Anaconda3/envs/GeoTable/Library/share/gmt
  version: 6.4.0

This is related to the comment at GenericMappingTools/pygmt#2728 (comment) by @seisman.

Compared to the images of the other remote datasets, the image of the GSHHG Global Earth Mask remote dataset has a fancy frame and annotations (please see https://www.generic-mapping-tools.org/remote-datasets/). This does not look well and is inconsistent. Thus it is suggested to update this image with a version without the annotated fancy frame.

Currently, the scripts/remote_map_check.sh script only works for earth dataset.

This feature request is to add a section to the remote dataset documentation for DCW including either a link to Federico's PDF files or a table showing the outlines of the regions.

Link to the zip file with the batch scripts for generated pdfs - GenericMappingTools/dcw-gmt#31 (comment)

PDFs of the collections - GenericMappingTools/dcw-gmt#31 (comment)

This is a request that Kurt Schwehr made in response to our twitter post announcing the datasets to add license info so that folks know if the datasets are okay for a particular use.

https://twitter.com/gmt_dev/status/1493343171601915908

There are horizontal lines in offshore area. I think it is an error in the processing.

This is 15s without those lines.

gmt grdimage @earth_relief_30s -png Earth_30s -I -RFK+r1 -Baf
gmt grdimage @earth_relief_15s -png Earth_15s -I -RFK+r1 -Baf