dlr-eoc / ukis-csmask Goto Github PK

onnxruntime>1.9 requires to specifically set execution providers. Running ukis-csmask with latest version of onnxruntime therefore fails with the following error:

ValueError: This ORT build has ['TensorrtExecutionProvider', 'CUDAExecutionProvider', 'CPUExecutionProvider'] enabled. Since ORT 1.9, you are required to explicitly set the providers parameter when instanti...

simply adding the available providers to the session would resolve this. here's an example how we could do this:

sess = onnxruntime.InferenceSession(
    str(Path(__file__).parent) + "/model.onnx", providers=onnxruntime.get_available_providers()
)

After cloud segmentation, invalid pixels are currently buffered using binary dilation in order to reduce the effect of fuzzy cloud and shadow boundaries. The number of iterations on how often dilation is applied is currently hard-coded to 4.

In some cases, however, it would be required to increase this value, if still some cloud disturbance is present that affects e.g. land cover classification models.

It would therefore be great, if the user could set the iteration parameter via an invalid_buffer argument.

We check for band names to ensure that the user provides the correct spectral bands to the package. Currently, we use a custom naming scheme for this "Red", "Green", "Blue", "NIR", "SWIR1", "SWIR2". While this works and is well documented in ukis-csmask, it is not in line with current best practices regarding STAC metadata.

Therefore, suggest to update the naming scheme for band to match the common-band-names scheme developed for the eo extension of STAC metadata.

We could still ensure backwards compatibility, but stay up-to-date with STAC, if we use "red", "green", "blue", "nir", "swir16", "swir22" instead.

Hi, I'm curious what image shapes the algorithm is expected to perform well on. Is there a range of image dimensions (rows, columns) of 30m pixels that the algorithm is limited to?

For example, is it expected to perform on a 16x16 image, in the same way that it would perform on a 512x512 image?

I see from the paper that the model was trained on 256x256 images, and assume the receptive field is likely constrained when image dimensions are too small, but the algorithm did not throw any errors when I used a very small image size (7x7 pixels).

Thank you!

We increasingly have the need to run cloud and cloud-shadow segmentation on satellite images that only have R-G-B-NIR spectral bands. This is particularly relevant when dealing with (very) high-resolution satellite images (e.g. RapidEye, Spot, GeoEye, Planet Scope, etc.).

ukis-csmask currently only supports images with six spectral bands (including SWIR1 and SWIR 2) as it has specifically been trained for Landsat and Sentinel-2 like data. In the original publication the 6 band model showed better performance compared to a 4 band model. However, the 4 band model still produced acceptable results.

Therefore, I suggest to implement the 4 band model as option into ukis-csmask for situations when we only have images with limited availability of spectral bands.

We currently support the default CPU and GPU providers for onnxruntime. It would be great to have the option to use OpenVino instead of the default CPU provider when running ukis-csmask on CPU. This can have significant performance gains over the default CPU.

intra_op_num_threads sets the number of threads used to parallelize the execution within nodes. Default is 0 to let onnxruntime choose. As reported in this onnxruntime issue, it may be required to manually set the number of threads in order to avoid issues with thread affinity.

Therefore, suggest to add onnxruntime session options to uski-csmask and expose the relevant options, so that the user can manually adjust them according to their specific hardware setup. Keep default values of onnxruntime to ensure backwards compatibility.

Is your feature request related to a problem? Please describe.
The Sentinel-2 L2A and Landsat Collection 2 products are processed to surface reflectance, while the current version of the UKIS cloud and shadow masking model was trained on top of atmosphere reflectance. Although the model can still be applied to surface reflectance imagery, this is not what it was trained on. It begs the question if a model trained on surface reflectance would perform better than the current TOA model.

Describe the solution you'd like
A UKIS CSMask trained on surface reflectance from the Sentinel 2 L2A and Landsat Collection 2 (4/5/7/8/9) archive.

The current implementation can be run on CPU or on CUDA enabled GPU. The current default requirements use onnxruntime, which runs ukis-csmask on CPU.

It would be great to have an install tag for ukis-csmask to enable GPU. The only thing that we would have to modify is to use onnxruntime-gpu dependency instead.

Not sure how this is best solved with pip (e.g. two packages ukis-csmask and ukis-csmask-gpu). Any ideas, experiences, best practises welcome.

I think the repository and paper would fit there, more information here: https://paperswithcode.com/about