osgeonepal / geotile Goto Github PK

View Code? Open in Web Editor NEW

65.0 2.0 9.0 126 KB

GeoTile is the python library for tiling the geographic raster data (eg. Tiff etc)

Home Page: https://geotile.readthedocs.io

License: MIT License

Python 100.00%

python raster spatialdata tiff tiling geotiler tiling-image gis map-tiles remote-sensing

geotile's People

Contributors

Stargazers

Watchers

Forkers

pankajkarman saadahmedjamal geethav93 sinaans richardburcher curtis18 kmkamilkhel

geotile's Issues

Data normalization

Need to implement the data normalization function.

Need to write function to create the numpy array

MemoryError: Unable to allocate ... GiB for an array with shape (num_features, 256, 256, 4) and data type `float64`

---------------------------------------------------------------------------
MemoryError                               Traceback (most recent call last)
[c:\Users\tek.kshetri\OneDrive](file:///C:/Users/tek.kshetri/OneDrive) - University of Calgary (1)\sand_mining\code\02. generate_tiles\imagery_&_mask_tiles_near_ratu.ipynb Cell 11 line 2
      [1](vscode-notebook-cell:/c%3A/Users/tek.kshetri/OneDrive%20-%20University%20of%20Calgary%20%281%29/sand_mining/code/02.%20generate_tiles/imagery_%26_mask_tiles_near_ratu.ipynb#X20sZmlsZQ%3D%3D?line=0) # tile_data = tile_data.astype(np.uint16)
----> [2](vscode-notebook-cell:/c%3A/Users/tek.kshetri/OneDrive%20-%20University%20of%20Calgary%20%281%29/sand_mining/code/02.%20generate_tiles/imagery_%26_mask_tiles_near_ratu.ipynb#X20sZmlsZQ%3D%3D?line=1) gt.save_numpy(f'{OUTPUT_TILES}/planet_near_ratu.npy')
      [3](vscode-notebook-cell:/c%3A/Users/tek.kshetri/OneDrive%20-%20University%20of%20Calgary%20%281%29/sand_mining/code/02.%20generate_tiles/imagery_%26_mask_tiles_near_ratu.ipynb#X20sZmlsZQ%3D%3D?line=2) # np.save(f'{OUTPUT_TILES}/planet_near_ratu.npy', tile_data)

File [c:\Users\tek.kshetri\.conda\envs\gee\lib\site-packages\geotile\GeoTile.py:573](file:///C:/Users/tek.kshetri/.conda/envs/gee/lib/site-packages/geotile/GeoTile.py:573), in GeoTile.save_numpy(self, file_name, dtype)
    570     self.tile_data = np.array(self.tile_data)
    572 # save the numpy file
--> 573 np.save(file_name, self.tile_data.astype(dtype))

MemoryError: Unable to allocate 73.9 GiB for an array with shape (37842, 256, 256, 4) and data type float64

gt.save_tile have issue

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
/mnt/d/tek/sm/code/prediction_merge.ipynb Cell 20 line 1
----> 1 gt.save_tiles(f'{PLACE_NAME}_S2_{YM}', prefix="tiles_")

File ~/miniconda3/envs/tf212/lib/python3.9/site-packages/geotile/GeoTile.py:470, in GeoTile.save_tiles(self, output_folder, prefix, suffix, image_format, dtype)
    467 meta.update({"count": wd.shape[0]})
    469 # save the tiles with new metadata
--> 470 with rio.open(tile_path, "w", **meta) as outds:
    471     outds.write(wd.astype(meta["dtype"]))

File ~/miniconda3/envs/tf212/lib/python3.9/site-packages/rasterio/env.py:451, in ensure_env_with_credentials.<locals>.wrapper(*args, **kwds)
    448     session = DummySession()
    450 with env_ctor(session=session):
--> 451     return f(*args, **kwds)

File ~/miniconda3/envs/tf212/lib/python3.9/site-packages/rasterio/__init__.py:314, in open(fp, mode, driver, width, height, count, crs, transform, dtype, nodata, sharing, **kwargs)
    312 writer = get_writer_for_driver(driver)
    313 if writer is not None:
--> 314     dataset = writer(
    315         path,
    316         mode,
    317         driver=driver,
    318         width=width,
    319         height=height,
    320         count=count,
    321         crs=crs,
    322         transform=transform,
    323         dtype=dtype,
    324         nodata=nodata,
    325         sharing=sharing,
    326         **kwargs
    327     )
    328 else:
    329     raise DriverCapabilityError(
    330         "Writer does not exist for driver: %s" % str(driver)
    331     )

File rasterio/_io.pyx:1502, in rasterio._io.DatasetWriterBase.__init__()

ValueError: Given nodata value, nan, is beyond the valid range of its data type, uint8.

conda installation of geotile in ubuntu 20.04

ModuleNotFoundError: No module named 'geotile'
I am getting this error while executing the example that has been mentioned in the git profile.

Please help.

Given nodata value, -32768.0, is beyond the valid range of its data type, uint8.

This error message is similar to #64 and #56. I think, we need to pass the nodata as a parameter.

File ~/miniconda3/envs/tf212/lib/python3.9/site-packages/geotile/GeoTile.py:476, in GeoTile.save_tiles(self, output_folder, prefix, suffix, image_format, dtype)
    473 meta.update({"count": wd.shape[0]})
    475 # save the tiles with new metadata
--> 476 with rio.open(tile_path, "w", **meta) as outds:
    477     outds.write(wd.astype(meta["dtype"]))

File ~/miniconda3/envs/tf212/lib/python3.9/site-packages/rasterio/env.py:451, in ensure_env_with_credentials.<locals>.wrapper(*args, **kwds)
    448     session = DummySession()
    450 with env_ctor(session=session):
--> 451     return f(*args, **kwds)

File ~/miniconda3/envs/tf212/lib/python3.9/site-packages/rasterio/__init__.py:314, in open(fp, mode, driver, width, height, count, crs, transform, dtype, nodata, sharing, **kwargs)
    312 writer = get_writer_for_driver(driver)
    313 if writer is not None:
--> 314     dataset = writer(
    315         path,
    316         mode,
    317         driver=driver,
    318         width=width,
    319         height=height,
    320         count=count,
    321         crs=crs,
    322         transform=transform,
    323         dtype=dtype,
    324         nodata=nodata,
    325         sharing=sharing,
    326         **kwargs
    327     )
    328 else:
    329     raise DriverCapabilityError(
    330         "Writer does not exist for driver: %s" % str(driver)
    331     )

File rasterio/_io.pyx:1502, in rasterio._io.DatasetWriterBase.__init__()

ValueError: Given nodata value, -32768.0, is beyond the valid range of its data type, uint8.

Auto API document is not working!

`gt.merge_tiles` method is not working as expected!

In merge_tiles method, the current implementation doesn't merge the tiles and generate the correct merged file. Please rewrite the function to fix this bug.

Need to implement data augmentation technique

stride_x, stride_y value of zero has some issue

The generate_raster_tiles method has issue with zero value of stride_x, stride_y

Data normalization based on z-score

Formula: z = (x - mean) / st.deviation
Normally, value lies between -3 to +3.

`gt.save_tiles` issue: TypeError: ufunc 'isnan' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[19], line 13
     11 gt.generate_tiles(save_tiles=False)
     12 gt.convert_nan_to_zero()
---> 13 gt.save_tiles(f"../../../data/tiles/mask_l8_{year}/{mask_location}", prefix=f'{mask_location}_l8_')

File ~/miniconda3/envs/tf212/lib/python3.9/site-packages/geotile/GeoTile.py:449, in GeoTile.save_tiles(self, output_folder, prefix, suffix, image_format, dtype)
    446 meta.update({"dtype": dtype or self.get_dtype(self.tile_data)})
    448 # solve nodata related issue #56
--> 449 if ((np.isnan(meta['nodata'])) and (meta['dtype'] in _int_dtypes)):
    450     meta.update({'nodata': None})
    452 # iterate through the offsets and windows_data and save the tiles

TypeError: ufunc 'isnan' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''

Tile Size Issue

In this line of code,
gt.generate_tiles(r'/path/to/output/folder', tile_x=256, tile_y=256, stride_x=256, stride_y=256)

i am not able to change the tile size to 64 . It takes 256 only...What can be the reason?

Thank you,

ModuleNotFoundError: No module named 'geotile'

Hi !

When I import geotile, following the readme from geotile import GeoTile gives me:
ModuleNotFoundError: No module named 'geotile'

I figured out that geotile is actually registered as geoTile so from geoTile import GeoTile works.

Thank you for your work by the way !

naming the tiles based on custom format

gt.generate_tiles() function need some custom name format option.

Need function to get the info about tiles

Filter the vectorize values based on the filter conditon

By default, the vectorize function generate the vector records for all possible values (for ex, in case of binary file, it generate polygon both for 0 and 1, which is annoying since most of the time we only need 1 value). In that case, we need to add one more parameter to filter out the vector files.

Fiona is not defined error in vectorize

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
/mnt/d/tek/sm/code/prediction_merge.ipynb Cell 25 line 4
      [1](vscode-notebook-cell://wsl%2Bubuntu/mnt/d/tek/sm/code/prediction_merge.ipynb#X31sdnNjb2RlLXJlbW90ZQ%3D%3D?line=0) from geotile.utils import vectorize 
      [2](vscode-notebook-cell://wsl%2Bubuntu/mnt/d/tek/sm/code/prediction_merge.ipynb#X31sdnNjb2RlLXJlbW90ZQ%3D%3D?line=1) import fiona
----> [4](vscode-notebook-cell://wsl%2Bubuntu/mnt/d/tek/sm/code/prediction_merge.ipynb#X31sdnNjb2RlLXJlbW90ZQ%3D%3D?line=3) vectorize('../data/img&pred/Hetauda_S2_2016_12_pred.tif', '../data/pred_shp/Dang_S2_2016_11_pred.shp')

File [~/miniconda3/envs/tf212/lib/python3.9/site-packages/geotile/utils.py:100](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/mnt/d/tek/sm/code/~/miniconda3/envs/tf212/lib/python3.9/site-packages/geotile/utils.py:100), in vectorize(input_raster, output_file, band, mask)
     [97](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/mnt/d/tek/sm/code/~/miniconda3/envs/tf212/lib/python3.9/site-packages/geotile/utils.py:97) shapes = rio.features.shapes(raster, transform=src.transform, mask=mask)
     [99](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/mnt/d/tek/sm/code/~/miniconda3/envs/tf212/lib/python3.9/site-packages/geotile/utils.py:99) # Save the vectorized raster
--> [100](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/mnt/d/tek/sm/code/~/miniconda3/envs/tf212/lib/python3.9/site-packages/geotile/utils.py:100) with fiona.open(output_file, 'w', crs=src.crs, driver='ESRI Shapefile', schema={'geometry': 'Polygon', 'properties': [('value', 'int')]}) as dst:
    [101](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/mnt/d/tek/sm/code/~/miniconda3/envs/tf212/lib/python3.9/site-packages/geotile/utils.py:101)     for geom, value in shapes:
    [102](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/mnt/d/tek/sm/code/~/miniconda3/envs/tf212/lib/python3.9/site-packages/geotile/utils.py:102)         dst.write({'geometry': geom, 'properties': {'value': value}})

NameError: name 'fiona' is not defined

Mask function generating RGB image instead of single band image.

from the Geotile documentation on how to generate the mask, the code used was as follows:

from geotile import GeoTile
gt = GeoTile('/path/to/raster/file.tif')
 
# generate shapefile mask
gt.mask('/path/to/shapefile.shp', '/path/to/output/file.tif')

First, the user would have to initialize the geotile using a raster file, i.e., gt = GeoTile('/path/to/raster/file.tif') Second, the user would have to generate the mask by specifying the directory for the shapefile and the output directory for the resulting raster, i.e., gt.mask('/path/to/shapefile.shp', '/path/to/output/file.tif').

The documentation also states clearly that, "The output raster will have similar metadata as the input raster."

So I tried creating a mask using a sample shapefile and initializing the geotile using my 3-band RGB image, and the resulting mask was not binary but an RGB image.

From your tutorial, I saw that you initialized your geotile using an existing binary mask, and as such, you could get a binary mask when you masked your shapefile. In reality, however, there would not be an existing mask for a user to use to initialize their geotile. The only raster that would be available is the multiband image. Kindly take a look and fix the error.

This message from udemy course here: https://www.udemy.com/course/deep-learning-application-for-earth-observation/?referralCode=B4655A8A918826871A40

Rasterization issue: ValueError: Given nodata value, nan, is beyond the valid range of its data type, uint8.

File [.conda\envs\gee\lib\site-packages\geotile\GeoTile.py:605](file://.conda/envs/gee/lib/site-packages/geotile/GeoTile.py:605), in GeoTile.rasterization(self, input_vector, out_path, value_col, **kwargs)
    602 meta.update({'count': 1, "dtype": "uint8"})
    604 # write the output raster
--> 605 with rio.open(out_path, 'w', **meta) as outds:
    606     outds.write(mask)

File [.conda\envs\gee\lib\site-packages\rasterio\env.py:451](file://.conda/envs/gee/lib/site-packages/rasterio/env.py:451), in ensure_env_with_credentials.<locals>.wrapper(*args, **kwds)
    448     session = DummySession()
    450 with env_ctor(session=session):
--> 451     return f(*args, **kwds)

File [.conda\envs\gee\lib\site-packages\rasterio\__init__.py:314](file://.conda/envs/gee/lib/site-packages/rasterio/__init__.py:314), in open(fp, mode, driver, width, height, count, crs, transform, dtype, nodata, sharing, **kwargs)
    312 writer = get_writer_for_driver(driver)
    313 if writer is not None:
--> 314     dataset = writer(
    315         path,
...
    331     )

File rasterio\_io.pyx:1502, in rasterio._io.DatasetWriterBase.__init__()

ValueError: Given nodata value, nan, is beyond the valid range of its data type, uint8.

mosaic raster issue: Bounds and transform are inconsistent

---------------------------------------------------------------------------
WindowError                               Traceback (most recent call last)
/mnt/d/tek/sm/code/prediction_merge.ipynb Cell 24 line 3
      1 from geotile import mosaic
----> 3 mosaic(f'{PLACE_NAME}_S2_{YM}', f'../data/img&pred/s2_rivers/pred/{PLACE_NAME}_S2_{YM}_pred.tif')

File ~/miniconda3/envs/tf212/lib/python3.9/site-packages/geotile/utils.py:49, in mosaic(input_folder, output_file, image_format, **kwargs)
     46     src_files_to_mosaic.append(src)
     48 # Merge the rasters
---> 49 mosaic, out_trans = merge(src_files_to_mosaic, **kwargs)
     51 # update the metadata
     52 meta = src.meta.copy()

File ~/miniconda3/envs/tf212/lib/python3.9/site-packages/rasterio/merge.py:344, in merge(datasets, bounds, res, nodata, dtype, precision, indexes, output_count, resampling, method, target_aligned_pixels, dst_path, dst_kwds)
    341 src_window = windows.from_bounds(int_w, int_s, int_e, int_n, src.transform)
    343 # 3. Compute the destination window
--> 344 dst_window = windows.from_bounds(
    345     int_w, int_s, int_e, int_n, output_transform
    346 )
    348 # 4. Read data in source window into temp
    349 src_window_rnd_shp = src_window.round_lengths()

File ~/miniconda3/envs/tf212/lib/python3.9/site-packages/rasterio/windows.py:324, in from_bounds(left, bottom, right, top, transform, height, width, precision)
    322     raise WindowError("Bounds and transform are inconsistent")
    323 if (bottom - top) / transform.e < 0:
--> 324     raise WindowError("Bounds and transform are inconsistent")
    326 rows, cols = rowcol(
    327     transform,
    328     [left, right, right, left],
    329     [top, top, bottom, bottom],
    330     op=float,
    331 )
    332 row_start, row_stop = min(rows), max(rows)

WindowError: Bounds and transform are inconsistent

Need function to construct the single prediction raster from tiles

Need function to Convert no data value to zero values

Similar function, convert_nan_to_zero().

Error in mosaicing

Hi,
I have used your library to break sat images in to tiles and used deep learning model for prediction of the individual tiles. Post prediction i want to to mosiaing to the same large image. Not able to do so.

Small filename issue when generating tiles

I run into a small issue regarding the file names when I generate tiles like below:
gt.generate_tiles(r'../output/', tile_x=256, tile_y=256, stride_x=128, stride_y=128)

The resulting file names will have an additional dot appended to them like:
/output_tiles_file/tile_7424_6144..tif'

Expected behavior just add a single dot for the file format extension itself:
/output_tiles_file/tile_7424_6144.tif'

Need to optimize the generate_tiles function.

I tested two different code,

Frist version:

for i, s2 in enumerate(s2_imagery):
    print(i, s2)

    # if i==1:
    #     break
    
    # get the imagery location name (eg. pokhara, kathmandu, janakpur etc.)
    imagery_location = os.path.basename(s2).split('_')[0] 

    # get year
    year = os.path.basename(s2).split('_')[-2]

    # read imagery
    s2_gt = GeoTile(s2)
    s2_gt.generate_tiles(save_tiles=False)
    s2_gt.convert_nan_to_zero()
    s2_gt.normalize_tiles()

    # # save tiles
    s2_gt.save_tiles(f"{OUTPUT_TILES}_{year}/{imagery_location}", prefix=f'{imagery_location}_s2_')
    
    print(imagery_location)

    # # # get the mask file based on each locations (eg. rapti, tinau, ratu)
    for j, mask in enumerate(s2_masks):

        if i == j:
            print('Imagery location: ', imagery_location)
            print('Mask location: ', os.path.basename(mask).split('_')[0])

            s2_gt.rasterization(mask, f"{OUTPUT_MASK}/{imagery_location}_{year}_s2_mask.tif")

Second version:

for i, mask in enumerate(mask_files):
    gt = GeoTile(mask)

    mask_location = os.path.basename(mask).split('_')[0]
    year = os.path.basename(mask).split('_')[1]

    # generate the mask file based on each binary output
    gt.generate_tiles(f"../../../data/tiles/mask_rivers_{year}/{mask_location}", prefix=f'{mask_location}_s2_')

The first version tool only 2 min 27 second to run the code however, the second version took 6 min 39 second. My guess is there is something going wrong with parameter save_tiles=True.