nasa-gibs / onearth Goto Github PK

I'm trying to get tile by long, lat and zoom by formula:

row = int((90 - lat) / 180 * 2 ** zoom)
col = int((180 + lon) / 180 * 2 ** zoom)

but it doesn't work.
Here is related question https://gis.stackexchange.com/questions/239496/how-to-convert-long-lat-to-a-tile-of-nasa-gibs-wgs-84-lat-lon-geographic

Got this error for the xml (
http://map1b.vis.earthdata.nasa.gov/colormaps/AIRS_RelativeHumidity_400hPa_Day.xml) with oe_generate_legend.py

   <ColorMapEntry rgb="161,0,3" transparent="False" nodata="False" sourceValue="[98,99)" value="[98,99)" label="98 – 99 %" ref="0"/>
    <ColorMapEntry rgb="151,0,3" transparent="False" nodata="False" sourceValue="[99,100)" value="[99,100)" label="99 – 100 %" ref="0"/>
    <ColorMapEntry rgb="142,0,3" transparent="False" nodata="False" sourceValue="[100]" value="[100]" label="100 %" ref="0"/>
</ColorMap>

Traceback (most recent call last):
  File "/Users/russ/Rails/airs_rails/lib/assets/oe_generate_legend.py", line 952, in <module>
    generate_legend(colormaps, output_location, options.format, options.orientation)
  File "/Users/russ/Rails/airs_rails/lib/assets/oe_generate_legend.py", line 467, in generate_legend
    increment = (float(colormap_entries[idx].value) - float(colormap_entries[idx-1].value))
ValueError: could not convert string to float: [99,100)

Even though the catch seems to be reporting on the [99,100) it is definitely on the [100] piece when it explodes.

Changed the else clause at line 463:

else:
   increment = (float(colormap_entries[idx].value) - float(colormap_entries[idx-1].value))
   bounds.append(float(colormap_entries[idx].value)+ increment)
   ticks.append(float(colormap_entries[idx].value) + increment/2)
   ticklabels.append(float(colormap_entries[idx].value))

and removed the lines referencing increment, so that the clause is now just

   ticklabels.append(float(colormap_entries[idx].value))

Which works, and the resulting image looks good, but not sure if I have broken something else...

Is there any roadmap available for OE2 ? or a release date ?

I'm really interested about this new version that seems to have a better approach for cloud configuration and deployment. 2.1.1 have been released since 10th of September and is tagged "testing only", could we have more informations about that ? (Why it is test only ? What have been tested and which part is considered unstable ?)

dupe

error ,during the test example-MORCR143ARDY;

https://github.com/nasa-gibs/onearth/blob/master/src/vectorgen/README.md

When describing the contents of the configuration file, it is said that "output_format" value should be either "ESRI Shapefile" or "MRF-MVT".

Second value is wrong, it should be "MVT-MRF" (oe_vectorgen.py line 416 will "fail")

This seems likely to be related to my version of python (2.7), but I am getting the following error when processing xml.

Reading color map: http://map1b.vis.earthdata.nasa.gov/colormaps/AIRS_Dust_Score.xml
Accessing URL http://map1b.vis.earthdata.nasa.gov/colormaps/AIRS_Dust_Score.xml
Traceback (most recent call last):
File "/Users/russ/Rails/airs_rails/lib/assets/oe_generate_legend.py", line 932, in
colormap_elements = parse_colormaps(colormap_location, options.verbose)
File "/Users/russ/Rails/airs_rails/lib/assets/oe_generate_legend.py", line 202, in parse_colormaps
tree=ET.fromstring(dom.toxml())
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xml/etree/ElementTree.py", line 1300, in XML
parser.feed(text)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/xml/etree/ElementTree.py", line 1640, in feed
self._parser.Parse(data, 0)
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2013' in position 217: ordinal not in range(128)

This is happening on files loaded via the local store, or via url.

Update the handling of color palettes during ingest to utilize the layer's SLD.

Forgive my ignorance here, but I am wondering: how does gibs/onearth compare with geoserver?

Is development here driven by something geoserver lacks or is there a more fundamental difference?

I tried to do an RPM rebuild using gdal 1.11.1, but it failed. Something must have changed as there is now a python install that is not respecting DESTDIR or the previous patch is no longer working.

make[4]: Entering directory `/home/jmcgann/onearth/build/rpmbuild/BUILD/gibs-gdal-1.11.1/build/gdal/swig/python'
python setup.py install 
running install
error: can't create or remove files in install directory

The following error occurred while trying to add or remove files in the
installation directory:

    [Errno 13] Permission denied: '/usr/lib64/python2.6/site-packages/test-easy-install-30739.write-test'

I use mrfgen to generate PPNG layers from GeoTIFF. I'm grouping imagery on a 6h range and I noticed that when mrf_merge is set to True, the generated layer loose the nodata value.

My geotiff file is paletted and the nodata value is 255, I tried to force nodata with vrtnodata set to 255 but this doesn't changed the generated MRF. Adding -a_nodata to gdal_merge command solved the issue :

From gdal_merge documentation:

-a_nodata output_nodata_value:
(GDAL >= 1.9.0) Assign a specified nodata value to output bands.

Trying to pull mrfgen.py but missing import oe_utils

I'm trying to generate a vector layer with oe_vectorgen, the generation from geojson to shapefile with oe_vectorgen is going well but once i'm trying to configure the layer with oe_configure_layer it results in an error:

Traceback (most recent call last):
  File "/usr/bin/oe_configure_layer", line 2439, in <module>
    if not static and len(timeElements)>0:
NameError: name 'timeElements' is not defined

I've took a quick look to oe_configure_layer script and it seems that in my case timeElements is never initialized when reaching this line:

I noticed that timeElements is not initialized when time is handled for vector layer and this layer is not static:

This is my layer config file:

<LayerConfiguration>
 <Identifier>test1</Identifier>
 <Title>test1</Title>
 <FileNamePrefix>test1</FileNamePrefix>
 <Projection>EPSG:4326</Projection>
 <EnvironmentConfig>/etc/onearth/config/conf/environment_geographic.xml</EnvironmentConfig>
 <ArchiveLocation static="False" year="True" subdaily="True">/usr/share/onearth/shapefiles/test1</ArchiveLocation>
 <VectorType>POINT</VectorType>
 <VectorStyleFile>/usr/share/onearth/styles/test1.txt</VectorStyleFile>
 <Time>DETECT/PT15M</Time>
</LayerConfiguration>

test1 layer is a set of 25 points transformed into a shapefile and placed in a directory under shapefiles>test1>2016>test1TTTTTTTTTTTTT_.*

This is the mrfgen configuration used:

<?xml version="1.0" encoding="UTF-8"?>
<vectorgen_configuration>
 <date_of_data>20160419</date_of_data>
 <time_of_data>120000</time_of_data>
 <parameter_name>test1</parameter_name>
 <input_dir>source_files</input_dir>
 <output_dir>output_dir</output_dir>
 <working_dir>working_dir</working_dir>
 <logfile_dir>logfile_dir</logfile_dir>
 <output_name>{$parameter_name}%Y%j%H%M%S_</output_name>
 <output_format>ESRI Shapefile</output_format>
 <target_epsg>4326</target_epsg>
</vectorgen_configuration>

Hi,
I'm playing with this amazing project and following the guide I'm trying to convert a geotiff to MRF with mrfgen. The problem is that the output always contains empty ppg.
Here two examples:

PS C:\Users\fernando marino\IdeaProjects\copeEarthTools\sampleData> gdalinfo .\descending-warp.tif
Driver: GTiff/GeoTIFF
Files: .\descending-warp.tif
Size is 1213, 1083
Coordinate System is:
GEOGCS["WGS 84",
    DATUM["WGS_1984",
        SPHEROID["WGS 84",6378137,298.257223563,
            AUTHORITY["EPSG","7030"]],
        AUTHORITY["EPSG","6326"]],
    PRIMEM["Greenwich",0],
    UNIT["degree",0.0174532925199433],
    AUTHORITY["EPSG","4326"]]
Origin = (5.995173495983380,47.509431455090372)
Pixel Size = (0.011543701808463,-0.011543701808463)
Metadata:
  AREA_OR_POINT=Area
Image Structure Metadata:
  COMPRESSION=LZW
  INTERLEAVE=BAND
Corner Coordinates:
Upper Left  (   5.9951735,  47.5094315) (  5d59'42.62"E, 47d30'33.95"N)
Lower Left  (   5.9951735,  35.0076024) (  5d59'42.62"E, 35d 0'27.37"N)
Upper Right (  19.9976838,  47.5094315) ( 19d59'51.66"E, 47d30'33.95"N)
Lower Right (  19.9976838,  35.0076024) ( 19d59'51.66"E, 35d 0'27.37"N)
Center      (  12.9964286,  41.2585169) ( 12d59'47.14"E, 41d15'30.66"N)
Band 1 Block=1213x1 Type=Float32, ColorInterp=Gray
  NoData Value=nan

with mrf config:

<?xml version="1.0" encoding="UTF-8"?>
<mrfgen_configuration>
 <date_of_data>20041201</date_of_data>
 <parameter_name>lst_descending</parameter_name>
 <input_files>
  <file>/data/source_images/descending-warp.tif</file>
 </input_files>
 <input_dir>source_images</input_dir> 
 <output_dir>output_dir</output_dir>
 <working_dir>working_dir</working_dir>
 <mrf_empty_tile_filename>/usr/share/onearth/empty_tiles/Blank_RGBA_512.png</mrf_empty_tile_filename>
 <vrtnodata>0</vrtnodata>
 <mrf_blocksize>256</mrf_blocksize>
 <mrf_compression_type>PPNG</mrf_compression_type>
 <outsize>1213 1083</outsize>
 <source_epsg>4326</source_epsg>
 <target_epsg>3857</target_epsg>
 <overview_levels>2 4 8 16 32 64 128 256 512 1024</overview_levels>
 <overview_resampling>nearest</overview_resampling>
 <mrf_nocopy>false</mrf_nocopy>
</mrfgen_configuration>

A second example is the following, with a plain RGB geotiff in epsg:4326:

[root@33d416546a1c data]# gdalinfo source_images/lee_waves_t.tif 
Driver: GTiff/GeoTIFF
Files: source_images/lee_waves_t.tif
Size is 43786, 29780
Coordinate System is:
GEOGCS["WGS 84",
    DATUM["WGS_1984",
        SPHEROID["WGS 84",6378137,298.257223563,
            AUTHORITY["EPSG","7030"]],
        AUTHORITY["EPSG","6326"]],
    PRIMEM["Greenwich",0],
    UNIT["degree",0.0174532925199433],
    AUTHORITY["EPSG","4326"]]
Origin = (-6.532602040508253,59.538346928975820)
Pixel Size = (0.000124997129700,-0.000124997129700)
Metadata:
  AREA_OR_POINT=Area
Image Structure Metadata:
  COMPRESSION=JPEG
  INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left  (  -6.5326020,  59.5383469) (  6d31'57.37"W, 59d32'18.05"N)
Lower Left  (  -6.5326020,  55.8159324) (  6d31'57.37"W, 55d48'57.36"N)
Upper Right (  -1.0594777,  59.5383469) (  1d 3'34.12"W, 59d32'18.05"N)
Lower Right (  -1.0594777,  55.8159324) (  1d 3'34.12"W, 55d48'57.36"N)
Center      (  -3.7960399,  57.6771397) (  3d47'45.74"W, 57d40'37.70"N)
Band 1 Block=43786x16 Type=Byte, ColorInterp=Red
Band 2 Block=43786x16 Type=Byte, ColorInterp=Green
Band 3 Block=43786x16 Type=Byte, ColorInterp=Blue

mrf config:

cat mrf_configs/lee.xml

<?xml version="1.0" encoding="UTF-8"?>
<mrfgen_configuration>
 <date_of_data>20041201</date_of_data>
 <parameter_name>lee_waves</parameter_name>
 <input_files>
  <file>/data/source_images/lee_waves_t.tif</file>
 </input_files>
 <input_dir>source_images</input_dir> 
 <output_dir>output_dir</output_dir>
 <working_dir>working_dir</working_dir>
 <mrf_empty_tile_filename>/usr/share/onearth/empty_tiles/Blank_RGBA_512.png</mrf_empty_tile_filename>
 <vrtnodata>0</vrtnodata>
 <mrf_blocksize>256</mrf_blocksize>
 <mrf_compression_type>PPNG</mrf_compression_type>
 <outsize>43786 29780</outsize>
 <source_epsg>4326</source_epsg>
 <target_epsg>3857</target_epsg>
 <overview_levels>2 4 8 16 32 64 128 256 512 1024</overview_levels>
 <overview_resampling>nearest</overview_resampling>
 <mrf_nocopy>false</mrf_nocopy>
</mrfgen_configuration>

I suppose there is something wrong with my data or my configuration, but I can't understand what.
I can also provide you the full geotiff if you need it.

Thanks for your help.

Fernando

I've got a problem in trying to set up a WMS subdaily layer. I'm experimenting doing this by altering one of the layers provided in the demo, namely MYR4ODLOLLDY_global_10km. I've built the code on a RHEL6 system.

First, the following works fine. This is taken from 'epsg4326.map':

LAYER
NAME "MYR4ODLOLLDY_global_10km"
TYPE RASTER
STATUS ON
VALIDATION
"default_MYR4ODLOLLDY_global_10km_TIME" "TTTTTTT"
"MYR4ODLOLLDY_global_10km_TIME" "^[0-9]{7}$"
"default_MYR4ODLOLLDY_global_10km_YEAR" "YYYY"
"MYR4ODLOLLDY_global_10km_YEAR" "^[0-9]{4}$"
END
METADATA
"wms_title" "MYR4ODLOLLDY_global_10km"
"wms_extent" "-180 -90 180 90"
"wms_timeextent" "2014-10-04/2014-10-04/P1D"
"wms_timedefault" "2014-10-04"
END
DATA "/data/local/batsc/Work/WebService/onearth_setup/data/EPSG4326/MYR4ODLOLLDY_global_10km//%MYR4ODLOLLDY_global_10km_YEAR%/MYR4ODLOLLDY%MYR4ODLOLLDY_global_10km_TIME%_.mrf"
PROJECTION
"init=epsg:4326"
END
END

and the following URL, with the TIME parameter specified, retrieves an image successfully:

http://localhost:7777/onearth/wms/wms.cgi?SERVICE=WMS&VERSION=1.3.0&REQUEST=GetMap&FORMAT=image%2Fpng&TRANSPARENT=true&LAYERS=MYR4ODLOLLDY_global_10km&CRS=EPSG%3A4326&STYLES=&WIDTH=1536&HEIGHT=768&BBOX=-135%2C-270%2C135%2C270&TIME=2014-10-04

For information (which may or may not indicate if something is wrong with my setup), if I specify a TIME in the URL outside of the 'wms_timeextent' (e.g. 2014-10-03) I get the following error:

msWMSLoadGetMapParams(): WMS server error. Unable to access -- corrupt, empty or missing file

Now, I have copied the MYR4ODLOLLDY_global_10km data and given it a subdaily timestamp of 12UTC, i.e.:

batsc@localhost EPSG4326 $ ls -R MYR4ODLOLLDY_global_10km_test
MYR4ODLOLLDY_global_10km_test:
2014 YYYY

MYR4ODLOLLDY_global_10km_test/2014:
MYR4ODLOLLDY2014277120000_.idx MYR4ODLOLLDY2014277120000_.mrf MYR4ODLOLLDY2014277120000_.ppg

MYR4ODLOLLDY_global_10km_test/YYYY:
MYR4ODLOLLDYTTTTTTTTTTTTT_.idx MYR4ODLOLLDYTTTTTTTTTTTTT_.mrf MYR4ODLOLLDYTTTTTTTTTTTTT_.ppg

and I've specified this test layer in the map file as:

LAYER
NAME "MYR4ODLOLLDY_global_10km_test"
TYPE RASTER
STATUS ON
VALIDATION
"default_MYR4ODLOLLDY_global_10km_test_TIME" "TTTTTTTTTTTTT"
"MYR4ODLOLLDY_global_10km_test_TIME" "^[0-9]{13}$"
"default_MYR4ODLOLLDY_global_10km_test_YEAR" "YYYY"
"MYR4ODLOLLDY_global_10km_test_YEAR" "^[0-9]{4}$"
END
METADATA
"wms_title" "MYR4ODLOLLDY_global_10km_test"
"wms_extent" "-180 -90 180 90"
"wms_timeextent" "2014-10-04T12:00:00Z/2014-10-04T12:00:00Z/P1D"
"wms_timedefault" "2014-10-04T12:00:00Z"
END
DATA "/data/local/batsc/Work/WebService/onearth_setup/data/EPSG4326/MYR4ODLOLLDY_global_10km_test//%MYR4ODLOLLDY_global_10km_test_YEAR%/MYR4ODLOLLDY%MYR4ODLOLLDY_global_10km_test_TIME%_.mrf"
PROJECTION
"init=epsg:4326"
END
END

When I use the following URL with an appropriate TIME parameter:

http://localhost:7777/onearth/wms/wms.cgi?SERVICE=WMS&VERSION=1.3.0&REQUEST=GetMap&FORMAT=image%2Fpng&TRANSPARENT=true&LAYERS=MYR4ODLOLLDY_global_10km_test&CRS=EPSG%3A4326&STYLES=&WIDTH=1536&HEIGHT=768&BBOX=-135%2C-270%2C135%2C270&TIME=2014-10-04T12%3A00%3A00Z

I get the following response:

Invalid TIME format, must be YYYY-MM-DD or YYYY-MM-DDThh:mm:ssZ

Specifying just TIME=2014-10-04 leads to the msWMSLoadGetMapParams(): WMS server error. Unable to access -- corrupt, empty or missing file error.

I've tried various things to see if I can get any clues as to what may be wrong but I'm out of ideas. I'm afraid I don't know C otherwise I'd play with the source code to output what it thinks is the time parameter specified and why it might be invalid.

Is there anything here that hints at what might be wrong? Any advice would be greatly appreciated.

I've setup a WFS endpoint with onearth but I've encountered some trouble:

• I have to specify the absolute path of the map file in my query otherwise wfs.cgi returns msCGILoadMap(): Web application error. CGI variable "map" is not set.. Is this a misconfiguration of my WFS endpoint ? Or is there at least any way to specify only the relative path to MapfileDir ? Relative path returns Unable to access file. (epsg4326.map) and it doesn't point to the DefaultMapfile without map variable.

This is the WFS endpoint configuration :

# WFS Endpoint
<Directory "/usr/share/onearth/wfs/epsg4326">
    MapfileDir /usr/share/onearth/mapserver/
    DefaultMapfile epsg4326.map
    Options Indexes FollowSymLinks ExecCGI
    AllowOverride None
    Order allow,deny
    Allow from all
    AddHandler cgi-script .cgi
    Header set Access-Control-Allow-Origin *
</Directory>

• TIME is not managed by WFS endpoint ?? Time variable doesn't have any effect on the query result
  /wfs/epsg4326/wfs.cgi?request=GetFeature&service=WFS&version=2.0.0&typeNames=ms:test1&map=/usr/share/onearth/mapserver/epsg4326.map&time=2016-04-19T12:00:00Z

After reading some documentation I just found that WFS endpoint doesn't have a TIME management like WMS :(
Is it possible to manage TIME with WFS properties and filter ? any solution for time dependent features ?
My use case is for windpseed features (Serve GFS through WFS features for exemple)

First of all thanks for OnEarth! A bit complicated to setup at first, but it works like a charm once we get through the configuration.

I was wondering if we could serve tiles from Object Storage like S3 and Swift (i.e. set archive location to a bucket like s3://archive-repo/)? Is this even feasible or it would involve too much delay ?

This would be awesome for scaling

Currently we are generating png's from the xml, but the labels are black, making them invisible on a black background. Is it possible to specify that you want the legend to appear white, not sure if this could be specified in the xml already and I just don't have it, or if this is not possible currently and I will need to hack the code to get it in there.

'model' tag should be optional in the layer config XML